opensource.com

Subscribe to opensource.com feed
Updated: 2 hours 23 min ago

What's new with Awk?

Sat, 10/01/2022 - 15:00
What's new with Awk? Jim Hall Sat, 10/01/2022 - 03:00

Awk is a powerful scripting tool that makes it easy to process text. Awk scripts use a pattern-action syntax, where Awk performs an action for every line in a file that matches a pattern. This provides a flexible yet powerful scripting language to deal with text. For example, the one-line Awk script /error/ {print $1, $2, $3} will print the first three space-delimited fields for any line that contains the word error.

While we also have the GNU variant of Awk, called Gawk, the original Awk remains under development. Recently, Brian Kernighan started a project to add Unicode support to Awk. I met with Brian to ask about the origins of Awk and his recent development work on Awk.

Jim Hall: Awk is a great tool to parse and process text. How did it start?

Brian Kernighan: The most direct influence was a tool that Marc Rochkind developed while working on the Programmer's Workbench system at Bell Labs. As I remember it now, Marc's program took a list of regular expressions and created a C program that would read an input file. Whenever the program found a match for one of the regular expressions, it printed the matching line. It was designed for creating error checking to run over log files from telephone operations data. It was such a neat idea—Awk is just a generalization.

Jim: AWK stands for the three of you who created it: Al Aho, Peter Weinberger, and Brian Kernighan. How did the three of you design and create Awk?

Brian: Al was interested in regular expressions and had recently implemented egrep, which provided a very efficient lazy-evaluation technique for a much bigger class of regular expressions than what grep provided. That gave us a syntax and working code.

Peter had been interested in databases, and as part of that he had some interest in report generation, like the RPG language that IBM provided. And I had been trying to figure out some kind of editing system that made it possible to handle strings and numbers with more or less equal ease.

We explored designs, but not for a long time. I think Al may have provided the basic pattern-action paradigm, but that was implicit in a variety of existing tools, like grep, the stream editor sed, and in the language tools YACC and Lex that we used for implementation. Naturally, the action language had to be C-like.

Jim: How was Awk first used at Bell Labs? When was Awk first adopted into Unix?

Brian: Awk was created in 1977, so it was part of 7th-edition Unix, which I think appeared in about 1979. I wouldn't say it was adopted, so much as it was just another program included because it was there. People picked it up very quickly, and we soon had users all over the Labs. People wrote much bigger programs than we had ever anticipated, too, even tens of thousands of lines, which was amazing. But for some kinds of applications, the language was a good match.

Jim: Has Awk changed over the years, or is Awk today more or less the same Awk from 1977?

Brian: Overall, it's been pretty stable, but there have been a fair number of small things, mostly to keep up with at least the core parts of Gawk. Examples include things like functions to do case conversion, shorthands for some kinds of regular expressions, or special filenames like /dev/stderr. Internally, there's been a lot of work to replace fixed-size arrays with arrays that grow. Arnold Robbins, who maintains Gawk, has also been incredibly helpful with Awk, providing good advice, testing, code, and help with Git.

Jim: You're currently adding Unicode support to Awk. This is one of those projects that seems obvious when you hear it, because Unicode is everywhere, but not every program supports it yet. Tell us about your project to add Unicode to Awk.

Brian: It's been sort of embarrassing for a while now that Awk only handled 8-bit input, though in fairness it predates Unicode by 10 or 20 years. Gawk, the GNU version, has handled Unicode properly for quite a while, so it's good to be up to date and compatible.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles

Jim: How big of a project is adding Unicode support? Did this require many changes to the source code?

Brian: I haven't counted, but it's probably 200 or 300 lines, primarily concentrated in either the regular expression recognizer or in the various built-in functions that have to operate in characters, not bytes, for Unicode input.

Jim: How far along are you in adding Unicode to Awk?

Brian: There's a branch of the code at GitHub that's pretty up to date. It's been tested, but there's always room for more testing.

One thing to mention: It handles UTF-8 input and output, but for Unicode code points, which are not the same thing as Unicode graphemes. This distinction is important but technically very complicated, at least as I understand it. As a simple example, a letter with an accent could be represented as two code points (letter and accent) or as a single character (grapheme). Doing this right, whatever that means, is very hard.

Jim: In a Computerphile video, you mention adding support for comma-separated values (CSV) parsing to Awk. How is that project going?

Brian: While I had my hands in the code again, I did add support for CSV input, since that's another bit of the language that was always clunky. I haven't done anything for CSV output, since that's easy to do with a couple of short functions, but maybe that should be revisited.

Jim: What kinds of things do you use Awk for in your day-to-day work?

Brian: Everything. Pretty much anything that fiddles text is a target for Awk. Certainly, the Awk program I use most is a simple one to make all lines in a text document the same length. I probably used it 100 times while writing answers to your questions.

Jim: What's the coolest (or most unusual) thing you have used Awk to do?

Brian: A long time ago, I wrote a C++ program that converted Awk programs into C++ that looked as close to Awk as I could manage, by doing things like overloading brackets for associative arrays. It was never used, but it was a fun exercise.

Further reading

Brian Kernighan discusses the scripting tool Awk, from its creation to current work on Unicode support.

Image by:

Jonas Leupe on Unsplash

Programming What to read next Talking digital with Brian Kernighan Learn awk by coding a "guess the number" game This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

How I dock my Linux laptop

Fri, 09/30/2022 - 15:00
How I dock my Linux laptop Don Watkins Fri, 09/30/2022 - 03:00

Not that long ago, docking a laptop was a new idea to me. Since then, I've set up two different laptops for docking, and I've been more than satisfied with how this configuration works. This article describes how I set up the laptop docking stations and gives tips and guidance for anyone considering the same.

How I discovered docking

I began 2020 with one laptop: a Darter Pro purchased from System76 the year before. It had been more than adequate, and I enjoyed the freedom of movement that toting a laptop gave me. Then, of course, everything changed.

Suddenly, I was at home all the time. Instead of an occasional video conference, remote meetings and get-togethers became the norm. I grew tired of the little square images of colleagues on the Darter Pro's 15.6-inch display, and the included 256 GB Non-Volatile Memory express (NVMe) drive was a limited amount of storage for my needs.

I purchased an Intel NUC (Next Unit of Computing) kit and a 27-inch LCD display, rearranged my office with a table, and settled into this new paradigm. Later in the year, I built another, more powerful NUC with an i7 processor, 32 GB of RAM, and a terabyte NVMe drive to accommodate video games and the video converted from old family 8mm movies and VHS tapes.

Now I had a laptop and a desktop (both running Linux distributions, of course). I regularly met with colleagues and friends on a variety of video conferencing platforms, and I thoroughly enjoyed the increased desktop real estate afforded by the 27-inch display.

Fast-forward 18 months to summer 2022, when I discovered laptop docking stations.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Docking a Linux laptop

I didn't realize I could have docked my Darter Pro using its USB-C port. My appetite for new hardware had me pining for the new HP Dev One, which comes with Pop!_OS preinstalled, a Ryzen 7 processor, 16 GB of RAM, a terabyte NVMe drive, and two USB-C ports.

Opensource.com alumnus Jay LeCroix has an excellent video explaining how to use USB-C docking with Linux laptops. That's when I began to appreciate the utility of USB-C docking hardware. After watching the video and doing some additional research, I decided to purchase a USB-C docking station for the Darter Pro.

System76 has a list of community-recommended docks, including the Plugable UD-CA1A dock. The Plugable dock claims to work with all Intel and NVIDIA systems, so I decided to order a unit. The dock was easy to set up, and once connected to the Darter Pro, I had the best of both worlds. I had the freedom to move when I wanted to and the ability to dock if I wanted to. But I needed more storage space than the 256GB drive in the Darter Pro.

My success with the docking station led me to purchase the HP Dev ONE, eager for the increased storage and speed. When it arrived, I quickly set it up. I backed up all the files on the NUC and restored them on my new Dev ONE. The new laptop was easily connected to the docking station.

Docking the Dev ONE

The Dev ONE has two USB-C ports, and either one connects to the dock. It still amazes me that power and video pass through this single port. In addition to the two USB-A ports on my laptop, I now have the additional ports on the docking station. The Plugable dock comes with a headphone jack, microphone jack, 4K high-definition multimedia interface (HDMI) output, one USB-C port on the rear, three USB-A 3.0 ports in the front, two USB-A 2.0 ports on the back, and a Gigabit Ethernet port. My new laptop has a webcam, but I plug an external webcam into one of the rear USB ports to accommodate video conferencing when I'm docked.

I enjoy the flexibility and power that docking has added to my workflow. I use video conferencing daily, and when I do, I'm docked and connected to a 27-inch display. I love how easy it is to transition from one setup to another. If you're using a laptop with a USB-C port, I recommend looking into a good docking station. It's well worth it.

Or are you already using a dock with your Linux laptop? What are your USB-C docking station recommendations? Be sure to share your experience in the comments below.

Docking a Linux laptop offers flexibility and a better video experience.

Image by:

Opensource.com

Linux Hardware What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

How Podman packaging works on Linux

Thu, 09/29/2022 - 15:00
How Podman packaging works on Linux Lokesh Mandvekar Thu, 09/29/2022 - 03:00

Over the past few months, the Podman project has been reworking its process for generating Debian and Ubuntu packages. This article outlines the past and present of the Debian packaging work done by the Podman project team. Please note that this article does not refer to the official Debian and Ubuntu packages that Reinhard Tartler and team created and maintain.

Debian build process

Long story short, the typical Debian build process involves "Debianizing" an upstream repository. First, a debian subdirectory containing packaging metadata and any necessary patches is added to the upstream repo. Then the dpkg-buildpackage command is run to generate the .deb packages.

Older Debian build process for Podman

Previously, the Debian packages for Podman were generated using this "Debianization" process. A debian directory containing the packaging metadata was added to the Podman source in a separate fork. That fork got rebased for each new upstream Podman release.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Issues with the Debian build process (for an RPM packager)

While a simple rebase would often work, that was not always the case. Usually, the Podman source itself would require patching to make things work for multiple Debian and Ubuntu versions, leading to rebase failures. And failures in a rebase meant failures in automated tasks. Which, in turn, caused a lot of frustration.

This same frustration led our team to retire the Debian packages in the past. When Podman v3.4 officially made its way into Debian 11 and Ubuntu 22.04 LTS (thanks to the amazing Reinhard Tartler), we thought the Podman project could say goodbye to Debian package maintenance.

But that wasn't meant to be. Both Debian and Ubuntu are rather conservative in their package update policies, especially in their release and LTS versions. As a result, many Podman users on Debian-based distributions would be stuck with v3.4 for quite a while, perhaps the entire lifetime of the distro version. While users can often install the latest packages from Debian's experimental repository, that's not necessarily convenient for everyone. As a result, many Debian-based users asked the Podman project for newer packages.

If we were to resurrect the Podman project's own Debian packages, we needed the packaging format to be easy to maintain and debug for RPM packagers and also easy to automate, which meant no frequent failures with rebases and patches.

OBS + Debbuild

The debbuild tool, created by Neal Gompa and others, is a set of RPM packaging macros allowing packagers to build Debian packages using Fedora's packaging sources. Conveniently, debbuild packages can easily be added as dependencies to a project hosted on openSUSE's Open Build Service infrastructure.

Here's a snippet of how debbuild support is enabled for Ubuntu 22.04 on the OBS Stable Kubic repository, maintained by the Podman project:

 
   
   
    x86_64
    s390x
    armv7l
    aarch64
 

The complete configuration file is available here.

In addition to enabling debbuild packages as dependencies, the Fedora packaging sources must be updated with rules to modify the build process for Debian and Ubuntu environments.

Here's a snippet of how it's done for Podman:

%if "%{_vendor}" == "debbuild"
Packager: Podman Debbuild Maintainers
License: ASL-2.0+ and BSD and ISC and MIT and MPLv2.0
Release: 0%{?dist}
%else
License: ASL 2.0 and BSD and ISC and MIT and MPLv2.0
Release: %autorelease
ExclusiveArch: %{golang_arches}
%endif

The " %{_vendor}" == "debbuild" conditional is used in many other places throughout the spec file. For example, in this code sample, it specifies different sets of dependencies and build steps for Fedora and Debian. Also, debbuild allows conditionalizing Debian and Ubuntu versions using the macros {debian} and {ubuntu}, which are familiar to RPM packagers.

You can find the updated RPM spec file with all the debbuild changes here.

These two pieces together produce successful Debian package builds on the OBS Unstable Kubic repository.

Using debbuild also ensures that packaging metadata lives in its own separate repository, implying no patching or rebasing hassles with upstream Podman sources.

Usability

At this time, packages are available for Ubuntu 22.04, Debian Testing, and Debian Unstable. We're in talks with the OBS infrastructure maintainers to adjust the Debian 11 and Ubuntu 20.04 build environments, after which we'll also have successful builds for those two environments.

$ apt list podman
Listing... Done
podman/unknown,now 4:4.2.0-0ubuntu22.04+obs55.1 amd64 [installed]
podman/unknown 4:4.2.0-0ubuntu22.04+obs55.1 arm64
podman/unknown 4:4.2.0-0ubuntu22.04+obs54.1 armhf
podman/unknown 4:4.2.0-0ubuntu22.04+obs54.1 s390x

Now, let's talk usability. These packages have been manually verified, and the Podman team has found them to satisfy typical use cases. Users can install these packages as they would any other DEB package. The repository first needs to be enabled, and there are instructions on the Podman website. However, these packages are not Debian-approved. They haven't gone through the same quality assurance process as official Debian packages. These packages are currently not recommended for production use, and we urge you to exercise caution before proceeding with installation.

Wrap up

The debbuild project allows the Podman project team to generate Debian packages with a few additions to the Fedora packaging sources, making Debian packaging easier to maintain, debug, and automate. It also allows Debian and Ubuntu users to get the latest Podman at the same speed as Fedora users. Podman users on Debian and Ubuntu looking for the latest updates can use our Kubic unstable repository (ideally not on production environments just yet.)

We also highly recommend the debbuild and OBS setup to RPM packagers who must provide Debian and Ubuntu packages to their users. It's a diverse selection of tooling, but open source is all about working together.

Get a deep dive into Podman packages for Debian and Ubuntu using Fedora Sources, OBS, and Debbuild.

Image by:

Opensource.com

Linux Kubernetes What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Build an open source project using this essential advice

Wed, 09/28/2022 - 15:00
Build an open source project using this essential advice Bolaji Ayodeji Wed, 09/28/2022 - 03:00

Open source is a flourishing and beneficial ecosystem that publicly solves problems in communities and industries using software developed through a decentralized model and community contributions. Over the years, this ecosystem has grown in number and strength among hobbyists and professionals alike. It's mainstream now—even proprietary companies use open source to build software.

With the ecosystem booming, many developers want to get in and build new open source projects. The question is: How do you achieve that successfully?

This article will demystify the lifecycle and structure of open source projects. I want to give you an overview of what goes on inside an open source project and show you how to build a successful and sustainable project based on my personal experience.

A quick introduction to open source

The Open Source Initiative (OSI) provides a formal, detailed definition of open source, but Wikipedia provides a nice summary:

Open source software is computer software that is released under a license in which the copyright holder grants users the rights to use, study, change, and distribute the software and its source code to anyone and for any purpose.

Open source software is public code, usually on the internet, developed either collaboratively by multiple people or by one person. It's about collaborating with people from different regions, cultures, and technical backgrounds, often working remotely. This is why creating a project that welcomes everyone and enables different people to work together is essential.

The anatomy of an open source project

Like the human body, an open source project is made up of several structures that form the entire system. I think of them as two branches: the people (microscopic) and the documents (macroscopic).

Branch one: people

Generally, an open source project includes the following sets of people:

  • Creators: Those who created the project
  • Maintainers: Those who actively manage the entire project
  • Contributors: Those who contribute to the project (someone like you!)
  • User: Those who use the project, including developers and nontechnical customers
  • Working group: A collection of contributors split into domain-specific groups to focus on a discussion or activity around a specific subject area (such as documentation, onboarding, testing, DevOps, code reviews, performance, research, and so on)
  • Sponsor: Those who contribute financial support to the project

You need to consider each group in the list above as you prepare to build a new project. What plan do you have for each of them?

  • For maintainers, decide on the criteria you want to use to appoint them. Usually, an active contributor makes the best maintainer.
  • For users and contributors, you want to prepare solid documentation, an onboarding process, and everything else they need to succeed when working with your project.
  • For working groups, decide whether you need them and how your project may be logically split in the future.
  • Finally, for sponsors, you must provide enough data and information about your project to enable them to choose to sponsor you.

You don't need to have all of these figured out at the start of your project. However, it's wise to think about them at the early stages so you can build the right foundations to ensure that future additions stand firm and lead to a successful project.

More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources Branch two: documents

Open source projects usually include the following documents, usually in plain text or markdown format:

  • License: This legal document explains how and to what extent the project can be freely used, modified, and shared. A list of OSI-approved licenses is available on the OSI website. Without an explicit license, your project is not legally open source!
     
  • Code of conduct: This document outlines the rules, norms, acceptable practices, and responsibilities of anyone who decides to participate in the project in any way (including what happens when someone violates any of the rules). The Contributor Covenant is a good example and is open source (licensed under a Creative Commons license).
     
  • README: This file introduces your project to newcomers. On many Git hosting websites, such as GitLab, GitHub, and Codeberg, the README file is displayed under the initial file listing of a repository. It's common to feature documentation here, with links to other necessary documents.
     
  • Documentation: This is a file or directory containing all documentation resources for the project, including guides, API references, tutorials, and so on.
     
  • Contributing: Include a document explaining how to contribute to the project, including installation guides, configuration, and so on.
     
  • Security: Include a file explaining how to submit vulnerability reports or security issues.

Additionally, a project usually has web pages for issues, support, and collaboration.

Broadly, these include:

  • Issues or bug reports: A place where users can report bugs. This page also provides a place developers can go to assign themselves the task of fixing one or more of them.
     
  • Pull or merge requests: A place with proposed feature enhancements and solutions to bugs. These patches may be created by anyone, reviewed by the maintainers, then merged into the project's code.
     
  • Discussions: A place where maintainers, contributors, and users discuss an open source project. This may be a dedicated website or a forum within a collaborative coding site.

Most projects also have a communication channel in the form of an online chat for conversations and interactions between community members.

Licensing

Licensing is perhaps the easiest but most important criterion to consider before creating an open source project. A license defines the terms and conditions that allow the source code and other components of your project to be used, modified, and shared.

Licenses contain tons of legal jargon that many people don't fully understand. I use choosealicense.com, which helps you choose a license based on your intended community, your desire to get patches back from those using your code, or your willingness to allow people to use your code without sharing improvements they make to it.

Image by:

(Bolaji Ayodeji, CC BY-SA 4.0)

13 phases of creating an open source project

Now for the essential question: How do you start an open source software project?

Here is a list of what I consider the phases of an open source project.

  1. Brainstorm your idea, write a synopsis, and document it properly.
     
  2. Begin developing your idea. This usually involves figuring out the right tools and stacks to use, writing some code, version controlling the code, debugging, drinking some coffee, hanging around StackOverflow, using other open source projects, sleeping, and building something to solve a defined problem—or just for fun!
     
  3. Test the project locally, write some unit and integration tests as required, set up CI/CD pipelines as needed, create a staging branch (a test branch where you test the code live before merging into the main branch), and do anything else you need to deploy the project.
     
  4. Write good and effective documentation. This should cover what your project does, why it is useful, how to get started with it (usage, installation, configuration, contributing), and where people can get support.
     
  5. Ensure to document all code conventions you want to use. Enforce them with tools like linters, code formatters, Git hooks, and the commitizen command line utility.
     
  6. Choose the right license and create a README.
     
  7. Publish the project on the internet (you might have a private repository initially, and make it public at this step).
     
  8. Set up the processes for making releases and documenting changelogs (you can use tools like Changesets).
     
  9. Market the project to the world! You can make a post on social media, start a newsletter, share it with your friends privately, do a product hunt launch, live stream, or any other traditional marketing strategy you know.
     
  10. Seek funding support by using any of the available funding platforms, like Open Collective, GitHub Sponsors, Patreon, Buy me a Coffee, LiberaPay, and so on. When you create accounts with these platforms, add a link to it in your project's documentation and website.
     
  11. Build a community around your project.
     
  12. Consider introducing working groups to break your project's management into logical parts when required.
     
  13. Continuously implement new ideas that sustain the resources and people behind your project.

It's important to measure different parts of your project as you progress. This provides you with data you can use for evaluation and future growth strategies.

Now start a project!

I hope this article helps you move forward with that project you've been thinking about.

Feel free to use it as a guide and fill any gaps I missed as you build your awesome open source software project.

Use these steps for a solid foundation for your first—or next—project.

Image by:

Opensource.com

Community management What to read next New open source tool catalogs African language resources I got my first pull request merged! 10 tips for onboarding open source contributors This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

165+ JavaScript terms you need to know

Tue, 09/27/2022 - 15:00
165+ JavaScript terms you need to know Sachin Samal Tue, 09/27/2022 - 03:00

JavaScript is a rich language, with sometimes a seemingly overwhelming number of libraries and frameworks. With so many options available, it's sometimes useful to just look at the language itself and keep in mind its core components. This glossary covers the core JavaScript language, syntax, and functions.

JavaScript variables

var: The most used variable. Can be reassigned but only accessed within a function, meaning function scope. Variables defined with var move to the top when code is executed.

const: Cannot be reassigned and not accessible before they appear within the code, meaning block scope.

let: Similar to const with block scope, however, the let variable can be reassigned but not re-declared.

Data types

Numbers: var age = 33

Variables: var a

Text (strings): var a = "Sachin"

Operations: var b = 4 + 5 + 6

True or false statements: var a = true

Constant numbers: const PI = 3.14

Objects: var fullName = {firstName:"Sachin", lastName: "Samal"}

Objects

This is a simple example of objects in JavaScript. This object describe the variable car, and includes keys or properties such as make, model, and year are the object's property names. Each property has a value, such as Nissan, Altima, and 2022. A JavaScript object is a collection of properties with values, and it functions as a method.

var car = {
make:"Nissan",
model:"Altima",
year:2022,
};Comparison operators

==: Is equal to

===: Is equal value and equal type

!=: Is not equal

!==: Is not equal value or not equal type

>: Is greater than

<: Is less than

>=: Is greater than or equal to

<=: Is less than or equal to

?: Ternary operator

Logical operators

&&: Logical AND

||: Logical OR

!: Logical NOT

Output data

alert(): Output data in an alert box in the browser window

confirm(): Open up a yes/no dialog and return true/false depending on user click

console.log(): Write information to the browser console. Good for debugging.

document.write(): Write directly to the HTML document

prompt(): Create a dialog for user input

Array methods

Array: An object that can hold multiple values at once.

concat(): Join several arrays into one

indexOf(): Return the primitive value of the specified object

join(): Combine elements of an array into a single string and return the string

lastIndexOf(): Give the last position at which a given element appears in an array

pop(): Remove the last element of an array

push(): Add a new element at the end

reverse(): Sort elements in descending order

shift(): Remove the first element of an array

slice(): Pull a copy of a portion of an array into a new array

splice(): Add positions and elements in a specified way

toString(): Convert elements to strings

unshift(): Add a new element to the beginning

valueOf(): Return the first position at which a given element appears in an array

Programming and development Red Hat Developers Blog Programming cheat sheets Try for free: Red Hat Learning Subscription eBook: An introduction to programming with Bash Bash shell scripting cheat sheet eBook: Modernizing Enterprise Java JavaScript loops

Loops: Perform specific tasks repeatedly under applied conditions.

for (before loop; condition for loop; execute after loop) {
// what to do during the loop
}

for: Creates a conditional loop

while: Sets up conditions under which a loop executes at least once, as long as the specified condition is evaluated as true

do while: Similar to the while loop, it executes at least once and performs a check at the end to see if the condition is met. If it is, then it executes again

break: Stop and exit the cycle at certain conditions

continue: Skip parts of the cycle if certain conditions are met

if-else statements

An if statement executes the code within brackets as long as the condition in parentheses is true. Failing that, an optional else statement is executed instead.

if (condition) {
// do this if condition is met
} else {
// do this if condition is not met
}Strings String methods

charAt(): Return a character at a specified position inside a string

charCodeAt(): Give the Unicode of the character at that position

concat(): Concatenate (join) two or more strings into one

fromCharCode(): Return a string created from the specified sequence of UTF-16 code units

indexOf(): Provide the position of the first occurrence of a specified text within a string

lastIndexOf(): Same as indexOf() but with the last occurrence, searching backwards

match(): Retrieve the matches of a string against a search pattern

replace(): Find and replace specified text in a string

search(): Execute a search for a matching text and return its position

slice(): Extract a section of a string and return it as a new string

split(): Split a string object into an array of strings at a specified position

substr(): Extract a substring depended on a specified number of characters, similar to slice()

substring(): Can't accept negative indices, also similar to slice()

toLowerCase(): Convert strings to lower case

toUpperCase(): Convert strings to upper case

valueOf(): Return the primitive value (that has no properties or methods) of a string object

Number methods

toExponential(): Return a string with a rounded number written as exponential notation

toFixed(): Return the string of a number with a specified number of decimals

toPrecision(): String of a number written with a specified length

toString(): Return a number as a string

valueOf(): Return a number as a number

Math methods

abs(a): Return the absolute (positive) value of a

acos(x): Arccosine of x, in radians

asin(x): Arcsine of x, in radians

atan(x): Arctangent of x as a numeric value

atan2(y,x): Arctangent of the quotient of its arguments

ceil(a): Value of a rounded up to its nearest integer

cos(a): Cosine of a (x is in radians)

exp(a): Value of Ex

floor(a): Value of a rounded down to its nearest integer

log(a): Natural logarithm (base E) of a

max(a,b,c…,z): Return the number with the highest value

min(a,b,c…,z): Return the number with the lowest value

pow(a,b): a to the power of b

random(): Return a random number between 0 and 1

round(a): Value of a rounded to its nearest integer

sin(a): Sine of a (a is in radians)

sqrt(a): Square root of a

tan(a): Tangent of an angle

Dealing with dates in JavaScript Set dates

Date(): Create a new date object with the current date and time

Date(2022, 6, 22, 4, 22, 11, 0): Create a custom date object. The numbers represent year, month, day, hour, minutes, seconds, milliseconds. You can omit anything except for year and month.

Date("2022-07-29"): Date declaration as a string

Pull date and time values

getDate(): Day of the month as a number (1-31)

getDay(): Weekday as a number (0-6)

getFullYear(): Year as a four-digit number (yyyy)

getHours(): Hour (0-23)

getMilliseconds(): Millisecond (0-999)

getMinutes(): Minute (0-59)

getMonth(): Month as a number (0-11)

getSeconds(): Second (0-59)

getTime(): Milliseconds since January 1, 1970

getUTCDate(): Day (date) of the month in the specified date according to universal time (also available for day, month, full year, hours, minutes, etc.)

parse: Parse a string representation of a date and return the number of milliseconds since January 1, 1970

Set part of a date

setDate(): Set the day as a number (1-31)

setFullYear(): Set the year (optionally month and day)

setHours(): Set the hour (0-23)

setMilliseconds(): Set milliseconds (0-999)

setMinutes(): Set the minutes (0-59)

setMonth(): Set the month (0-11)

setSeconds(): Set the seconds (0-59)

setTime(): Set the time (milliseconds since January 1, 1970)

setUTCDate(): Set the day of the month for a specified date according to universal time (also available for day, month, full year, hours, minutes, etc.)

Dom mode Node methods

appendChild(): Add a new child node to an element as the last child node

cloneNode(): Clone an HTML element

compareDocumentPosition(): Compare the document position of two elements

getFeature(): Return an object which implements the APIs of a specified feature

hasAttributes(): Return true if an element has any attributes, otherwise false

hasChildNodes(): Return true if an element has any child nodes, otherwise false

insertBefore(): Insert a new child node before a specified, existing child node

isDefaultNamespace(): Return true if a specified namespaceURI is the default, otherwise false

isEqualNode(): Check if two elements are equal

isSameNode(): Check if two elements are the same node

isSupported(): Return true if a specified feature is supported on the element

lookupNamespaceURI(): Return the namespaceURI associated with a given node

normalize(): Join adjacent text nodes and removes empty text nodes in an element

removeChild(): Remove a child node from an element

replaceChild(): Replace a child node in an element

Element methods

getAttribute(): Return the specified attribute value of an element node

getAttributeNS(): Return string value of the attribute with the specified namespace and name

getAttributeNode(): Get the specified attribute node

getAttributeNodeNS(): Return the attribute node for the attribute with the given namespace and name

getElementsByTagName(): Provide a collection of all child elements with the specified tag name

getElementsByTagNameNS(): Return a live HTMLCollection of elements with a certain tag name belonging to the given namespace

hasAttribute(): Return true if an element has any attributes, otherwise false

hasAttributeNS(): Provide a true/false value indicating whether the current element in a given namespace has the specified attribute

removeAttribute(): Remove a specified attribute from an element

lookupPrefix(): Return a DOMString containing the prefix for a given namespaceURI, if present

removeAttributeNS(): Remove the specified attribute from an element within a certain namespace

removeAttributeNode(): Take away a specified attribute node and return the removed node

setAttribute(): Set or change the specified attribute to a specified value

setAttributeNS(): Add a new attribute or changes the value of an attribute with the given namespace and name

setAttributeNode(): Set or change the specified attribute node

setAttributeNodeNS(): Add a new namespaced attribute node to an element

JavaScript events Mouse

onclick: User clicks on an element

oncontextmenu: User right-clicks on an element to open a context menu

ondblclick: User double-clicks on an element

onmousedown: User presses a mouse button over an element

onmouseenter: Pointer moves onto an element

onmouseleave: Pointer moves out of an element

onmousemove: Pointer moves while it is over an element

onmouseover: Pointer moves onto an element or one of its children

setInterval(): Call a function or evaluates an expression at

oninput: User input on an element

onmouseup: User releases a mouse button while over an element

onmouseout: User moves the mouse pointer out of an element or one of its children

onerror: Happens when an error occurs while loading an external file

onloadeddata: Media data is loaded

onloadedmetadata: Metadata (like dimensions and duration) is loaded

onloadstart: Browser starts looking for specified media

onpause: Media is paused either by the user or automatically

onplay: Media is started or is no longer paused

onplaying: Media is playing after having been paused or stopped for buffering

onprogress: Browser is in the process of downloading the media

onratechange: Media play speed changes

onseeked: User finishes moving/skipping to a new position in the media

onseeking: User starts moving/skipping

onstalled: Browser tries to load the media, but it is not available

onsuspend — Browser is intentionally not loading media

ontimeupdate: Play position has changed (e.g., because of fast forward)

onvolumechange: Media volume has changed (including mute)

onwaiting: Media paused but expected to resume (for example, buffering)

Keep this JavaScript glossary bookmarked to reference variables, methods, strings, and more.

Image by:

Photo by Jen Wike Huger

JavaScript What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Get change alerts from any website with this open source tool

Tue, 09/27/2022 - 15:00
Get change alerts from any website with this open source tool Leigh Morresi Tue, 09/27/2022 - 03:00

The year was 2020, and news about COVID-19 came flooding in so quickly that everyone felt completely overwhelmed with similar news articles providing updates with varying degrees of accuracy.

But all I needed to know was when my official government guidelines changed. In the end, that's all that mattered to me.

Whether the concern is a pandemic or just the latest tech news, keeping ahead of changes in website content can be critical.

The changedetection.io project provides a simple yet highly capable, open source solution for website change detection and notification. It's easy to set up, and it can notify over 70 (and counting) different notification systems, such as Matrix, Mattermost, Nextcloud, Signal, Zulip, Home Assistant, email, and more. It also notifies proprietary applications like Discord, Office365, Reddit, Telegram, and many others.

But changedetection.io isn't just limited to watching web page content. You can also monitor XML and JSON feeds, and it will build an RSS feed of the websites that changed.

More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources

Thanks to its built-in JSON simple storage system, there's no need to set up complicated databases to receive and store information. You can run it as a Docker image or install it with pip. The project has an extensive wiki help section, and most common questions are covered there.

For sites using complex JavaScript, you can connect your changedetection.io installation to a Chromium or Chrome browser with the built-in Playwright content fetcher.

Once running, access the application in your browser (http://localhost:5000, by default). You can set a password in the Settings section if your computer can be reached from an outside network.

Image by:

(Leigh Morresi, CC BY-SA 4.0)

Submit the URL of a page you want to monitor. There are several settings related to how the page is filtered. For example, you more than likely do not want to know when a company's stock price listed in their site footer has changed, but you may want to know when they post a news article to their blog.

Monitor a site

Imagine you want to add your favorite website, Opensource.com, to be monitored. You only want to know when the main call-out article contains the word "python" and you want to be notified over Matrix.

To do this, begin with the visual-selector tool. (This requires the playwright browser interface to be connected.)

Image by:

(Leigh Morresi, CC BY-SA 4.0)

The visual-selector tool automatically calculates the best Xpath or CSS filter to target the content. Otherwise, you would get a lot of noise from the daily page updates.

Next, visit the Filters & Triggers tab.

Image by:

(Leigh Morresi, CC BY-SA 4.0)

In CSS/JSON/XPATH Filter field (the blue circle), you can see the automatically generated CSS filter from the previous step.

There are several useful filters available, such as Remove elements (good for removing noisy elements), Ignore text, Trigger/wait for text, and Block change-detection if text matches (used for waiting for some text to disappear, like "sold out").

In Trigger/wait for text (the red circle), type in the keyword you want to monitor for. (That's "python" in this example.)

The final step is in the Notifications tab, where you configure where you want to receive your notification. Below I added a Matrix room as the notification target, using the Matrix API.

Image by:

(Leigh Morresi, CC BY-SA 4.0)

The notification URL is in the format of matrixs://username:password@matrix.org/#/room/#room-name:matrix.org

However, t2Bot format is also supported. Here are more Matrix notification options.

And that's it! Now you'll receive a message over Matrix whenever the content changes.

There's more

There's so much more to changedetection.io. If you prefer calling a custom JSON API, you don't have to use an API for notifications (use jsons://). You can also create a custom HTTP request (POST and GET), execute JavaScript before checking (perhaps to pre-fill a username and password login field), and many more interesting features, with more to come.

Stop browsing the web and start watching the web instead!

Use changedetection.io to get alerts when a website makes changes or updates.

Tools Web development What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

OpenSSF: on a mission to improve security of open source software

Mon, 09/26/2022 - 15:00
OpenSSF: on a mission to improve security of open source software Gaurav Kamathe Mon, 09/26/2022 - 03:00

Open source software (OSS), once a niche segment of the development landscape, is now ubiquitous. This growth is fantastic for the open source community. However, as the usage of OSS increases, so do concerns about security. Especially in mission-critical applications— think medical devices, automobiles, space flight, and nuclear facilities—securing open source technology is of the utmost priority. No individual entity, whether developers, organizations, or governments, can single-handedly solve this problem. The best outcome is possible when all of them come together to collaborate.

The Open Source Security Foundation (OpenSSF) formed to facilitate this collaboration. OpenSSF is best described in its own words:

The OpenSSF is a cross-industry collaboration that brings together leaders to improve the security of open source software by building a broader community with targeted initiatives and best practices.

Vision

The technical vision of OpenSSF is to handle security proactively, by default. Developers are rightly at the center of this vision. OpenSSF seeks to empower developers to learn secure development practices and automatically receive guidance on them through the day-to-day tools they use. Researchers who identify security issues can send this information backward through the supply chain to someone who can rapidly address the issue. Auditors and regulators are encouraged to devise security policies that can be easily enforced via tooling, and community members provide information on the components they use and test regularly.

Mobilization plan

OpenSSF drafted a mobilization plan based on input from open source developers and leaders from US federal agencies. The result is a set of high-impact actions aimed at improving the resiliency and security of open source software. Based on this plan, 10 streams of investments have been identified, including security education, risk assessment, memory safety, and supply chain improvement. While discussion of these issues is widespread, OpenSSF is the platform that has collected and prioritized these concerns over others to ensure a secure open source ecosystem.

Working groups

Because the 10 streams of investments are quite diverse, OpenSSF is divided into multiple working groups. This strategy allows individual teams to focus on a specific area of expertise and move forward without getting bogged down with more general concerns. The working groups have something for everyone: Developers can contribute to security tooling, maintainers can handle software repositories, and others can contribute by educating developers on best practices, identifying metrics for open source projects, or identifying and securing the critical projects that form the core of the OSS ecosystem.

More on security The defensive coding guide 10 layers of Linux container security SELinux coloring book More security articles Industry participation

Multiple software vendors have become members of OpenSSF in their own capacity. These vendors are important players in the IT ecosystem, ranging from cloud service providers and operating system vendors to companies hosting OSS repositories, creating security tooling, creating computing hardware, and more. The benefit is getting inputs from a variety of sources that others might not be aware of and then collaboratively working on those issues.

Getting involved

There are a variety of ways to participate in the OpenSSF initiative based on your expertise and the amount of time you can set aside for it:

  • Sign up for their mailing list to follow the latest updates and discussions and update your calendar with OpenSSF meetings.
  • If you are looking for more interactive communication, consider joining their Slack channel.
  • Browse through their past meetings on their YouTube channel.
  • Organizations can consider becoming a member of OpenSSF.
  • Developers can quickly look up the GitHub repo for the software projects they are working on.
  • Most important, consider joining a working group of your choice and make a difference.
Conclusion

The security industry is growing and needs active participation from the open source community. If you are starting out or wish to specialize in security, OpenSSF provides a platform to work on the right problems in the security space under the guidance of experienced peers in security.

Developers, businesses, and government agencies are working together to ensure the security of open source software, and you can join them.

Image by:

Tumisu. CC0

Security and privacy What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

The story behind Joplin, the open source note-taking app

Mon, 09/26/2022 - 15:00
The story behind Joplin, the open source note-taking app Richard Chambers Mon, 09/26/2022 - 03:00

In this interview, I met up with Laurent Cozic, creator of the note-taking app, Joplin. Joplin was a winner of the 20i rewards, so I wanted to find out what makes it such a success, and how he achieved it.

Could you summarize what Joplin does?

Joplin is an open source note-taking app. It allows you to capture your thoughts and securely access them from any device.

Obviously, there are other note-taking apps out there—but apart from it being free to use, what makes it different?

The fact that it is open source is an important aspect for many of our users, because it means there is no vendor locking on the data, and that data can be easily exported and accessed in various ways.

We also focus on security and data privacy, in particular with the synchronization end-to-end encryption feature, and by being transparent about any connection that the application makes. We also work with security researchers to keep the app more secure.

Finally, Joplin can be customized in several different ways—through plugins, which can add new functionalities, and themes to customize the app appearance. We also expose a data API, which allows third-party applications to access Joplin data.

[ Related read 5 note-taking apps for Linux ]

It's a competitive market, so what inspired you to build it?

It happened organically. I started looking into it in 2016, as I was looking at existing commercial note-taking applications, and I didn't like that the notes, attachments, or tags could not easily be exported or manipulated by other tools.

This is probably due to vendor locking and partly a lack of motivation from the vendor since they have no incentive to help users move their data to other apps. There is also an issue with the fact that these companies usually will keep the notes in plain text, and that can potentially cause issues in terms of data privacy and security.

So I decided to start creating a simple mobile and terminal application with sync capabilities to have my notes easily accessible on my devices. Later the desktop app was created and the project grew from there.

Image by:

(Opensource.com, CC BY-SA 4.0)

How long did Joplin take to make?

I've been working on it on and off since 2016 but it wasn't full time. The past two years I've been focusing more on it.

What advice might you have for someone setting to create their own open source app?

Pick a project you use yourself and technologies you enjoy working with.

Managing an open source project can be difficult sometimes so there has to be this element of fun to make it worthwhile. Then I guess "release early, release often" applies here, so that you can gauge user's interest and whether it makes sense to spend time developing the project further.

How many people are involved in Joplin's development?

There are 3-4 people involved in the development. At the moment we also have six students working on the project as part of Google Summer of Code.

[ Also read Our journey to open source during Google Summer of Code ]

Lots of people create open source projects, yet Joplin has been a resounding success for you. Could you offer creators any tips on how to get noticed?

There's no simple formula and to be honest I don't think I could replicate the success in a different project! You've got to be passionate about what you're doing but also be rigorous, be organized, make steady progress, ensure the code quality remains high, and have a lot of test units to prevent regressions.

Also be open to the user feedback you receive, and try to improve the project based on it.

Once you've got all that, the rest is probably down to luck—if it turns out you're working on a project that interests a lot of people, things might work out well!

Once you get noticed, how do you keep that momentum going, if you don't have a traditional marketing budget?

I think it's about listening to the community around the project. For example I never planned to have a forum but someone suggested it on GitHub, so I made one and it became a great way to share ideas, discuss features, provide support, and so on. The community is generally welcoming of newcomers too, which creates a kind of virtuous circle.

Next to this, it's important to communicate regularly about the project.

We don't have a public roadmap, because the ETA for most features is generally "I don't know", but I try to communicate about coming features, new releases, and so on. We also communicate about important events, the Google Summer of Code in particular, or when we have the chance to win something like the 20i FOSS Awards.

Finally, very soon we'll have an in-person meetup in London, which is another way to keep in touch with the community and collaborators.

How does user feedback influence the roadmap?

Significantly. Contributors will often work on something simply because they need the feature. But next to this, we also keep track of the features that seem most important to users, based on what we read about on the forum and on the GitHub issue tracker.

For example, the mobile app is now high priority because we frequently hear from users that its limitations and issues are a problem to effectively use Joplin.

Image by:

(Opensource.com, CC BY-SA 4.0)

How do you keep up to date with the latest in dev and coding?

Mostly by reading Hacker News!

Do you have a personal favorite FOSS that you'd recommend?

Among the less well-known projects, SpeedCrunch is very good as a calculator. It has a lot of features and it's great how it keeps a history of all previous calculations.

I also use KeepassXC as a password manager. It has been improving steadily over the past few years.

Finally, Visual Studio Code is great as a cross-platform text editor.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles

I'd assumed that Joplin was named after Janis, but Wikipedia tells me it's Scott Joplin. What made you choose the name?

I wanted to name it "jot-it" at first but I think the name was already taken.

Since I was listening to Scott Joplin ragtime music a lot back then (I was pretty much obsessed with it), I decided to use his name.

I think the meaning of a product name is not too important, as long as the name itself is easy to write, pronounce, remember, and perhaps is associated with something positive (or at least nothing negative).

And I think "Joplin" ticks all these boxes.

Is there anything you can say about plans for Joplin? An exclusive tease of a new feature, perhaps?

As mentioned earlier, we are very keen to make improvements to the mobile app, both in terms of UX design and new features.

We're also looking at creating a "Plugin Store" to make it easier to browse and install plugins.

Thanks for your time Laurent— best of luck with the future of Joplin.

This interview was originally published on the 20i blog and has been republished with permission.

Laurent Cozic sat down with me to discuss how Joplin got started and what's next for the open source note-taking app.

Image by:

Opensource.com

Art and design What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Drop your database for PostgreSQL

Sat, 09/24/2022 - 15:00
Drop your database for PostgreSQL Seth Kenlon Sat, 09/24/2022 - 03:00

Databases are tools to store information in an organized but flexible way. A spreadsheet is essentially a database, but the constraints of a graphical application render most spreadsheet applications useless to programmers. With Edge and IoT devices becoming significant target platforms, developers need powerful but lightweight solutions for storing, processing, and querying large amounts of data. One of my favourite combinations is the PostgreSQL database and Lua bindings, but the possibilities are endless. Whatever language you use, Postgres is a great choice for a database, but you need to know some basics before adopting it.

Install Postgres

To install PostgreSQL on Linux, use your software repository. On Fedora, CentOS, Mageia, and similar:

$ sudo dnf install postgresql postgresql-server

On Debian, Linux Mint, Elementary, and similar:

$ sudo apt install postgresql postgresql-contrib

On macOS and Windows, download an installer from postgresql.org.

Setting up Postgres

Most distributions install the Postgres database without starting it, but provide you with a script or systemd service to help it start reliably. However, before you start PostgreSQL, you must create a database cluster.

Fedora

On Fedora, CentOS, or similar, there's a Postgres setup script provided in the Postgres package. Run this script for easy configuration:

$ sudo /usr/bin/postgresql-setup --initdb
[sudo] password:
 * Initializing database in '/var/lib/pgsql/data'
 * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.logDebian

On Debian-based distributions, setup is performed automatically by apt during installation.

Everything else

Finally, if you're running something else, then you can just use the toolchain provided by Postgres itself. The initdb command creates a database cluster, but you must run it as the postgres user, an identity you may temporarily assume using sudo:

$ sudo -u postgres \
"initdb -D /var/lib/pgsql/data \
--locale en_US.UTF-8 --auth md5 --pwprompt"Start Postgres

Now that a cluster exists, start the Postgres server using either the command provided to you in the output of initdb or with systemd:

$ sudo systemctl start postgresqlCreating a database user

To create a Postgres user, use the createuser command. The postgres user is the superuser of the Postgres install,

$ sudo -u postgres createuser --interactive --password bogus
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) y
Shall the new role be allowed to create more new roles? (y/n) n
Password:Create a database

To create a new database, use the createdb command. In this example, I create the database exampledb and assign ownership of it to the user bogus:

$ createdb exampledb --owner bogusInteracting with PostgreSQL

You can interact with a PostgreSQL database using the psql command. This command provides an interactive shell so you can view and update your databases. To connect to a database, specify the user and database you want to use:

$ psql --user bogus exampledb
psql (XX.Y)
Type "help" for help.

exampledb=>Create a table

Databases contain tables, which can be visualized as a spreadsheet. There's a series of rows (called records in a database) and columns. The intersection of a row and a column is called a field.

The Structured Query Language (SQL) is named after what it provides: A method to inquire about the contents of a database in a predictable and consistent syntax to receive useful results.

Currently, your database is empty, devoid of any tables. You can create a table with the CREATE query. It's useful to combine this with the IF NOT EXISTS statement, which prevents PostgreSQL from clobbering an existing table.

Before you createa table, think about what kind of data (the "data type" in SQL terminology) you anticipate the table to contain. In this example, I create a table with one column for a unique identifier and one column for some arbitrary text up to nine characters.

exampledb=> CREATE TABLE IF NOT EXISTS my_sample_table(
exampledb(> id SERIAL,
exampledb(> wordlist VARCHAR(9) NOT NULL
);

The SERIAL keyword isn't actually a data type. It's special notation in PostgreSQL that creates an auto-incrementing integer field. The VARCHAR keyword is a data type indicating a variable number of characters within a limit. In this code, I've specified a maximum of 9 characters. There are lots of data types in PostgreSQL, so refer to the project documentation for a list of options.

Insert data

You can populate your new table with some sample data by using the INSERT SQL keyword:

exampledb=> INSERT INTO my_sample_table (wordlist) VALUES ('Alice');
INSERT 0 1

Your data entry fails, should you attempt to put more than 9 characters into the wordlist field:

exampledb=> INSERT INTO my_sample_table (WORDLIST) VALUES ('Alexandria');
ERROR:  VALUE too long FOR TYPE CHARACTER VARYING(9)Alter a table or column

When you need to change a field definition, you use the ALTER SQL keyword. For instance, should you decide that a nine character limit for wordlist, you can increase its allowance by setting its data type:

exampledb=> ALTER TABLE my_sample_table
ALTER COLUMN wordlist SET DATA TYPE VARCHAR(10);
ALTER TABLE
exampledb=> INSERT INTO my_sample_table (WORDLIST) VALUES ('Alexandria');
INSERT 0 1View data in a table

SQL is a query language, so you view the contents of a database through queries. Queries can be simple, or it can involve joining complex relationships between several different tables. To see everything in a table, use the SELECT keyword on * (an asterisk is a wildcard):

exampledb=> SELECT * FROM my_sample_table;
 id |  wordlist
\----+------------
  1 | Alice
  2 | Bob
  3 | Alexandria
(3 ROWS)More data

PostgreSQL can handle a lot of data, but as with any database the key to success is how you design your database for storage and what you do with the data once you've got it stored. A relatively large public data set can be found on OECD.org, and using this you can try some advanced database techniques.

First, download the data as comma-separated values (CSV) and save the file as land-cover.csv in your Downloads folder.

Browse the data in a text editor or spreadsheet application to get an idea of what columns there are, and what kind of data each column contains. Look at the data carefully and keep an eye out for exceptions to an apparent rule. For instance, the COU column, containing a country code such as AUS for Australia and GRC for Greece, tends to be 3 characters until the oddity BRIICS.

Once you understand the data you're working with, you can prepare a Postgres database:

$ createdb landcoverdb --owner bogus
$ psql --user bogus landcoverdb
landcoverdb=> create table land_cover(
country_code varchar(6),
country_name varchar(76),
small_subnational_region_code varchar(5),
small_subnational_region_name varchar(14),
large_subnational_region_code varchar(17),
large_subnational_region_name varchar(44),
measure_code varchar(13),
measure_name varchar(29),
land_cover_class_code varchar(17),
land_cover_class_name varchar(19),
year_code integer,
year_value integer,
unit_code varchar(3),
unit_name varchar(17),
power_code integer,
power_name varchar(9),
reference_period_code varchar(1),
reference_period_name varchar(1),
value float(8),
flag_codes varchar(1),
flag_names varchar(1));Importing data

Postgres can import CSV data directly using the special metacommand \copy:

landcoverdb=> \copy land_cover from '~/land-cover.csv' with csv header delimiter ','
COPY 22113

That's 22,113 records imported. Seems like a good start!

More on edge computing Understanding edge computing Why Linux is critical to edge computing eBook: Running Kubernetes on your Raspberry Pi Download now: The automated enterprise eBook eBook: A practical guide to home automation using open source tools eBook: 7 examples of automation on the edge The latest on edge Querying data

A broad SELECT statement to see all columns of all 22,113 records is possible, and Postgres very nicely pipes the output to a screen pager so you can scroll through the output at a leisurely pace. However, using advanced SQL you can get some useful views of what's otherwise some pretty raw data.

landcoverdb=> SELECT
    lcm.country_name,
    lcm.year_value,
    SUM(lcm.value) sum_value
FROM land_cover lcm
JOIN (
    SELECT
        country_name,
        large_subnational_region_name,
        small_subnational_region_name,
        MAX(year_value) max_year_value
    FROM land_cover
    GROUP BY country_name,
        large_subnational_region_name,
        small_subnational_region_name
) AS lcmyv
ON
    lcm.country_name = lcmyv.country_name AND
    lcm.large_subnational_region_name = lcmyv.large_subnational_region_name AND
    lcm.small_subnational_region_name = lcmyv.small_subnational_region_name AND
    lcm.year_value = lcmyv.max_year_value
GROUP BY lcm.country_name,
    lcm.large_subnational_region_name,
    lcm.small_subnational_region_name,
    lcm.year_value
ORDER BY country_name,
    year_value;

Here's some sample output:

\---------------+------------+------------
 Afghanistan    |       2019 |  743.48425
 Albania        |       2019 |  128.82532
 Algeria        |       2019 |  2417.3281
 American Samoa |       2019 |   100.2007
 Andorra        |       2019 |  100.45613
 Angola         |       2019 |  1354.2192
 Anguilla       |       2019 | 100.078514
 Antarctica     |       2019 |  12561.907
[...]

SQL is a rich langauge, and so it's beyond the scope of this article. Read through the SQL code and see if you can modify it to provide a different set of data.

Open database

PostgreSQL is one of the great open source databases. With it, you can design repositories for structured data, and then use SQL to view it in different ways so you can gain fresh perspectives on that data. Postgres integrates with many languages, including Python, Lua, Groovy, Java, and more, so regardless of your toolset, you can probably make use of this excellent database.

Postgres is one of the most flexible databases available, and it's open source.

Image by:

Image by Mapbox Uncharted ERG, CC-BY 3.0 US

Databases Edge computing What to read next Install MariaDB or MySQL on Linux 3 ways to use PostgreSQL commands This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 6870 points (Correspondent) Vancouver, Canada

Seldom without a computer of some sort since graduating from the University of British Columbia in 1978, I have been a full-time Linux user since 2005, a full-time Solaris and SunOS user from 1986 through 2005, and UNIX System V user before that.

On the technical side of things, I have spent a great deal of my career as a consultant, doing data analysis and visualization; especially spatial data analysis. I have a substantial amount of related programming experience, using C, awk, Java, Python, PostgreSQL, PostGIS and lately Groovy. I'm looking at Julia with great interest. I have also built a few desktop and web-based applications, primarily in Java and lately in Grails with lots of JavaScript on the front end and PostgreSQL as my database of choice.

Aside from that, I spend a considerable amount of time writing proposals, technical reports and - of course - stuff on https://www.opensource.com.

User Attributes Correspondent Open Sourcerer People's Choice Award 100+ Contributions Club Emerging Contributor Award 2016 Author Comment Gardener Correspondent Columnist Contributor Club Register or Login to post a comment.

How to build a dynamic distributed database with DistSQL

Fri, 09/23/2022 - 15:00
How to build a dynamic distributed database with DistSQL Raigor Jiang Fri, 09/23/2022 - 03:00

Distributed databases are common for many reasons. They increase reliability, redundancy, and performance. Apache ShardingSphere is an open source framework that enables you to transform any database into a distributed database. Since the release of ShardingSphere 5.0.0, DistSQL (Distributed SQL) has provided dynamic management for the ShardingSphere ecosystem.

In this article, I demonstrate a data sharding scenario in which DistSQL's flexibility allows you to create a distributed database. At the same time, I show some syntax sugar to simplify operating procedures, allowing your potential users to choose their preferred syntax.

A series of DistSQL statements are run through practical cases to give you a complete set of practical DistSQL sharding management methods, which create and maintain distributed databases through dynamic management.

Image by:

(Jiang Longtao, CC BY-SA 4.0)

What is sharding?

In database terminology, sharding is the process of partitioning a table into separate entities. While the table data is directly related, it often exists on different physical database nodes or, at the very least, within separate logical partitions.

Practical case example

To follow along with this example, you must have these components in place, either in your lab or in your mind as you read this article:

  • Two sharding tables: t_order and t_order_item.
  • For both tables, database shards are carried out with the user_id field, and table shards with the order_id field.
  • The number of shards is two databases times three tables.
Image by:

(Jiang Longtao, CC BY-SA 4.0)

Set up the environment

1. Prepare a database (MySQL, MariaDB, PostgreSQL, or openGauss) instance for access. Create two new databases: demo_ds_0 and demo_ds_1.

2. Deploy Apache ShardingSphere-Proxy 5.1.2 and Apache ZooKeeper. ZooKeeper acts as a governance center and stores ShardingSphere metadata information.

3. Configure server.yaml in the Proxy conf directory as follows:

mode:
  type: Cluster
  repository:
    type: ZooKeeper
    props:
      namespace: governance_ds
      server-lists: localhost:2181 #ZooKeeper address
      retryIntervalMilliseconds: 500
      timeToLiveSeconds: 60
      maxRetries: 3
      operationTimeoutMilliseconds: 500
  overwrite: falserules:
 - !AUTHORITY
    users:
     - root@%:root

4. Start ShardingSphere-Proxy and connect it to Proxy using a client, for example:

$ mysql -h 127.0.0.1 -P 3307 -u root -p

5. Create a distributed database:

CREATE DATABASE sharding_db;USE sharding_db;Add storage resources

Next, add storage resources corresponding to the database:

ADD RESOURCE ds_0 (
    HOST=127.0.0.1,
    PORT=3306,
    DB=demo_ds_0,
    USER=root,
    PASSWORD=123456
), ds_1(
    HOST=127.0.0.1,
    PORT=3306,
    DB=demo_ds_1,
    USER=root,
    PASSWORD=123456
);

View the storage resources:

mysql> SHOW DATABASE RESOURCES\G;
******** 1. row ***************************
         name: ds_1
         type: MySQL
         host: 127.0.0.1
         port: 3306
           db: demo_ds_1
          -- Omit partial attributes
******** 2. row ***************************
         name: ds_0
         type: MySQL
         host: 127.0.0.1
         port: 3306
           db: demo_ds_0
          -- Omit partial attributes

Adding the optional \G switch to the query statement makes the output format easy to read.

More on data science What is data science? What is Python? How to become a data scientist Data scientist: A day in the life What is big data? Whitepaper: Data-intensive intelligent applications in a hybrid cloud blueprint MariaDB and MySQL cheat sheet Latest data science articles Create sharding rules

ShardingSphere's sharding rules support regular sharding and automatic sharding. Both sharding methods have the same effect. The difference is that the configuration of automatic sharding is more concise, while regular sharding is more flexible and independent.

Refer to the following links for more details on automatic sharding:

Next, it's time to adopt regular sharding and use the INLINE expression algorithm to implement the sharding scenarios described in the requirements.

Primary key generator

The primary key generator creates a secure and unique primary key for a data table in a distributed scenario. For details, refer to the document Distributed Primary Key.

1. Create a primary key generator:

CREATE SHARDING KEY GENERATOR snowflake_key_generator (
TYPE(NAME=SNOWFLAKE)
);

2. Query the primary key generator:

mysql> SHOW SHARDING KEY GENERATORS;
+-------------------------+-----------+-------+
| name                    | type      | props |
+-------------------------+-----------+-------+
| snowflake_key_generator | snowflake | {}    |
+-------------------------+-----------+-------+
1 row in set (0.01 sec)Sharding algorithm

1. Create a database sharding algorithm used by t_order and t_order_item in common:

-- Modulo 2 based on user_id in database sharding
CREATE SHARDING ALGORITHM database_inline (
TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="ds_${user_id % 2}"))
);

2. Create different table shards algorithms for t_order and t_order_item:

-- Modulo 3 based on order_id in table sharding
CREATE SHARDING ALGORITHM t_order_inline (
TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="t_order_${order_id % 3}"))
);
CREATE SHARDING ALGORITHM t_order_item_inline (
TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="t_order_item_${order_id % 3}"))
);

3. Query the sharding algorithm:

mysql> SHOW SHARDING ALGORITHMS;
+---------------------+--------+---------------------------------------------------+
| name                | type   | props                                             |
+---------------------+--------+---------------------------------------------------+
| database_inline     | inline | algorithm-expression=ds_${user_id % 2}            |
| t_order_inline      | inline | algorithm-expression=t_order_${order_id % 3}      |
| t_order_item_inline | inline | algorithm-expression=t_order_item_${order_id % 3} |
+---------------------+--------+---------------------------------------------------+
3 rows in set (0.00 sec)Create a default sharding strategy

The sharding strategy consists of a sharding key and sharding algorithm, which in this case is databaseStrategy and tableStrategy. Because t_order and t_order_item have the same database sharding field and sharding algorithm, create a default strategy to be used by all shard tables with no sharding strategy configured.

1. Create a default database sharding strategy:

CREATE DEFAULT SHARDING DATABASE STRATEGY (
TYPE=STANDARD,SHARDING_COLUMN=user_id,SHARDING_ALGORITHM=database_inline
);

2. Query default strategy:

mysql> SHOW DEFAULT SHARDING STRATEGY\G;
*************************** 1. row ***************************
                    name: TABLE
                    type: NONE
         sharding_column:
 sharding_algorithm_name:
 sharding_algorithm_type:
sharding_algorithm_props:
*************************** 2. row ***************************
                    name: DATABASE
                    type: STANDARD
         sharding_column: user_id
 sharding_algorithm_name: database_inline
 sharding_algorithm_type: inline
sharding_algorithm_props: {algorithm-expression=ds_${user_id % 2}}
2 rows in set (0.00 sec)

You have not configured the default table sharding strategy, so the default strategy of TABLE is NONE.

Set sharding rules

The primary key generator and sharding algorithm are both ready. Now you can create sharding rules. The method I demonstrate below is a little complicated and involves multiple steps. In the next section, I'll show you how to create sharding rules in just one step, but for now, witness how it's typically done.

First, define t_order:

CREATE SHARDING TABLE RULE t_order (
DATANODES("ds_${0..1}.t_order_${0..2}"),
TABLE_STRATEGY(TYPE=STANDARD,SHARDING_COLUMN=order_id,SHARDING_ALGORITHM=t_order_inline),
KEY_GENERATE_STRATEGY(COLUMN=order_id,KEY_GENERATOR=snowflake_key_generator)
);

Here is an explanation of the values found above:

  • DATANODES specifies the data nodes of shard tables.
  • TABLE_STRATEGY specifies the table strategy, among which SHARDING_ALGORITHM uses created sharding algorithm t_order_inline.
  • KEY_GENERATE_STRATEGY specifies the primary key generation strategy of the table. Skip this configuration if primary key generation is not required.

Next, define t_order_item:

CREATE SHARDING TABLE RULE t_order_item (
DATANODES("ds_${0..1}.t_order_item_${0..2}"),
TABLE_STRATEGY(TYPE=STANDARD,SHARDING_COLUMN=order_id,SHARDING_ALGORITHM=t_order_item_inline),
KEY_GENERATE_STRATEGY(COLUMN=order_item_id,KEY_GENERATOR=snowflake_key_generator)
);

Query the sharding rules to verify what you've created:

mysql> SHOW SHARDING TABLE RULES\G;
************************** 1. row ***************************
                           table: t_order
               actual_data_nodes: ds_${0..1}.t_order_${0..2}
             actual_data_sources:
          database_strategy_type: STANDARD
        database_sharding_column: user_id
database_sharding_algorithm_type: inline
database_sharding_algorithm_props: algorithm-expression=ds_${user_id % 2}
              table_strategy_type: STANDARD
            table_sharding_column: order_id
    table_sharding_algorithm_type: inline
   table_sharding_algorithm_props: algorithm-expression=t_order_${order_id % 3}
              key_generate_column: order_id
               key_generator_type: snowflake
              key_generator_props:
*************************** 2. row ***************************
                            table: t_order_item
                actual_data_nodes: ds_${0..1}.t_order_item_${0..2}
              actual_data_sources:
           database_strategy_type: STANDARD
         database_sharding_column: user_id
 database_sharding_algorithm_type: inline
database_sharding_algorithm_props: algorithm-expression=ds_${user_id % 2}
              table_strategy_type: STANDARD
            table_sharding_column: order_id
    table_sharding_algorithm_type: inline
   table_sharding_algorithm_props: algorithm-expression=t_order_item_${order_id % 3}
              key_generate_column: order_item_id
               key_generator_type: snowflake
              key_generator_props:
2 rows in set (0.00 sec)

This looks right so far. You have now configured the sharding rules for t_order and t_order_item.

You can skip the steps for creating the primary key generator, sharding algorithm, and default strategy, and complete the sharding rules in one step. Here's how to make it easier.

Sharding rule syntax

For instance, if you want to add a shard table called t_order_detail, you can create sharding rules as follows:

CREATE SHARDING TABLE RULE t_order_detail (
DATANODES("ds_${0..1}.t_order_detail_${0..1}"),
DATABASE_STRATEGY(TYPE=STANDARD,SHARDING_COLUMN=user_id,SHARDING_ALGORITHM(TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="ds_${user_id % 2}")))),
TABLE_STRATEGY(TYPE=STANDARD,SHARDING_COLUMN=order_id,SHARDING_ALGORITHM(TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="t_order_detail_${order_id % 3}")))),
KEY_GENERATE_STRATEGY(COLUMN=detail_id,TYPE(NAME=snowflake))
);

This statement specifies a database sharding strategy, table strategy, and primary key generation strategy, but it doesn't use existing algorithms. The DistSQL engine automatically uses the input expression to create an algorithm for the sharding rules of t_order_detail.

Now there's a primary key generator:

mysql> SHOW SHARDING KEY GENERATORS;
+--------------------------+-----------+-------+
| name                     | type      | props |
+--------------------------+-----------+-------+
| snowflake_key_generator  | snowflake | {}    |
| t_order_detail_snowflake | snowflake | {}    |
+--------------------------+-----------+-------+
2 rows in set (0.00 sec)

Display the sharding algorithm:

mysql> SHOW SHARDING ALGORITHMS;
+--------------------------------+--------+-----------------------------------------------------+
| name                           | type   | props                                               |
+--------------------------------+--------+-----------------------------------------------------+
| database_inline                | inline | algorithm-expression=ds_${user_id % 2}              |
| t_order_inline                 | inline | algorithm-expression=t_order_${order_id % 3}        |
| t_order_item_inline            | inline | algorithm-expression=t_order_item_${order_id % 3}   |
| t_order_detail_database_inline | inline | algorithm-expression=ds_${user_id % 2}              |
| t_order_detail_table_inline    | inline | algorithm-expression=t_order_detail_${order_id % 3} |
+--------------------------------+--------+-----------------------------------------------------+
5 rows in set (0.00 sec)

And finally, the sharding rules:

mysql> SHOW SHARDING TABLE RULES\G;
*************************** 1. row ***************************
                            table: t_order
                actual_data_nodes: ds_${0..1}.t_order_${0..2}
              actual_data_sources:
           database_strategy_type: STANDARD
         database_sharding_column: user_id
 database_sharding_algorithm_type: inline
database_sharding_algorithm_props: algorithm-expression=ds_${user_id % 2}
              table_strategy_type: STANDARD
            table_sharding_column: order_id
    table_sharding_algorithm_type: inline
   table_sharding_algorithm_props: algorithm-expression=t_order_${order_id % 3}
              key_generate_column: order_id
               key_generator_type: snowflake
              key_generator_props:
*************************** 2. row ***************************
                            table: t_order_item
                actual_data_nodes: ds_${0..1}.t_order_item_${0..2}
              actual_data_sources:
           database_strategy_type: STANDARD
         database_sharding_column: user_id
 database_sharding_algorithm_type: inline
database_sharding_algorithm_props: algorithm-expression=ds_${user_id % 2}
              table_strategy_type: STANDARD
            table_sharding_column: order_id
    table_sharding_algorithm_type: inline
   table_sharding_algorithm_props: algorithm-expression=t_order_item_${order_id % 3}
              key_generate_column: order_item_id
               key_generator_type: snowflake
              key_generator_props:
*************************** 3. row ***************************
                            table: t_order_detail
                actual_data_nodes: ds_${0..1}.t_order_detail_${0..1}
              actual_data_sources:
           database_strategy_type: STANDARD
         database_sharding_column: user_id
 database_sharding_algorithm_type: inline
database_sharding_algorithm_props: algorithm-expression=ds_${user_id % 2}
              table_strategy_type: STANDARD
            table_sharding_column: order_id
    table_sharding_algorithm_type: inline
   table_sharding_algorithm_props: algorithm-expression=t_order_detail_${order_id % 3}
              key_generate_column: detail_id
               key_generator_type: snowflake
              key_generator_props:
3 rows in set (0.01 sec)

In the CREATE SHARDING TABLE RULE statement, DATABASE_STRATEGY, TABLE_STRATEGY, and KEY_GENERATE_STRATEGY can reuse existing algorithms.

Alternatively, they can be defined quickly through syntax. The difference is that additional algorithm objects are created.

Configuration and verification

Once you have created the configuration verification rules, you can verify them in the following ways.

1. Check node distribution:

DistSQL provides SHOW SHARDING TABLE NODES for checking node distribution, and users can quickly learn the distribution of shard tables:

mysql> SHOW SHARDING TABLE NODES;
+----------------+------------------------------------------------------------------------------------------------------------------------------+
| name           | nodes                                                                                                                        |
+----------------+------------------------------------------------------------------------------------------------------------------------------+
| t_order        | ds_0.t_order_0, ds_0.t_order_1, ds_0.t_order_2, ds_1.t_order_0, ds_1.t_order_1, ds_1.t_order_2                               |
| t_order_item   | ds_0.t_order_item_0, ds_0.t_order_item_1, ds_0.t_order_item_2, ds_1.t_order_item_0, ds_1.t_order_item_1, ds_1.t_order_item_2 |
| t_order_detail | ds_0.t_order_detail_0, ds_0.t_order_detail_1, ds_1.t_order_detail_0, ds_1.t_order_detail_1                                   |
+----------------+------------------------------------------------------------------------------------------------------------------------------+
3 rows in set (0.01 sec)

mysql> SHOW SHARDING TABLE NODES t_order_item;
+--------------+------------------------------------------------------------------------------------------------------------------------------+
| name         | nodes                                                                                                                        |
+--------------+------------------------------------------------------------------------------------------------------------------------------+
| t_order_item | ds_0.t_order_item_0, ds_0.t_order_item_1, ds_0.t_order_item_2, ds_1.t_order_item_0, ds_1.t_order_item_1, ds_1.t_order_item_2 |
+--------------+------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

You can see that the node distribution of the shard table is consistent with what is described in the requirement.

SQL preview

Previewing SQL is also an easy way to verify configurations. Its syntax is PREVIEW SQL. First, make a query with no shard key, with all routes:

mysql> PREVIEW SELECT * FROM t_order;
+------------------+---------------------------------------------------------------------------------------------+
| data_source_name | actual_sql                                                                                  |
+------------------+---------------------------------------------------------------------------------------------+
| ds_0             | SELECT * FROM t_order_0 UNION ALL SELECT * FROM t_order_1 UNION ALL SELECT * FROM t_order_2 |
| ds_1             | SELECT * FROM t_order_0 UNION ALL SELECT * FROM t_order_1 UNION ALL SELECT * FROM t_order_2 |
+------------------+---------------------------------------------------------------------------------------------+
2 rows in set (0.13 sec)

mysql> PREVIEW SELECT * FROM t_order_item;
+------------------+------------------------------------------------------------------------------------------------------------+
| data_source_name | actual_sql                                                                                                 |
+------------------+------------------------------------------------------------------------------------------------------------+
| ds_0             | SELECT * FROM t_order_item_0 UNION ALL SELECT * FROM t_order_item_1 UNION ALL SELECT * FROM t_order_item_2 |
| ds_1             | SELECT * FROM t_order_item_0 UNION ALL SELECT * FROM t_order_item_1 UNION ALL SELECT * FROM t_order_item_2 |
+------------------+------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Now specify user_id in a query with a single database route:

mysql> PREVIEW SELECT * FROM t_order WHERE user_id = 1;
+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| data_source_name | actual_sql                                                                                                                                        |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| ds_1             | SELECT * FROM t_order_0 WHERE user_id = 1 UNION ALL SELECT * FROM t_order_1 WHERE user_id = 1 UNION ALL SELECT * FROM t_order_2 WHERE user_id = 1 |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.14 sec)

mysql> PREVIEW SELECT * FROM t_order_item WHERE user_id = 2;
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| data_source_name | actual_sql                                                                                                                                                       |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ds_0             | SELECT * FROM t_order_item_0 WHERE user_id = 2 UNION ALL SELECT * FROM t_order_item_1 WHERE user_id = 2 UNION ALL SELECT * FROM t_order_item_2 WHERE user_id = 2 |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Specify user_id and order_id with a single table route:

mysql> PREVIEW SELECT * FROM t_order WHERE user_id = 1 AND order_id = 1;
+------------------+------------------------------------------------------------+
| data_source_name | actual_sql                                                 |
+------------------+------------------------------------------------------------+
| ds_1             | SELECT * FROM t_order_1 WHERE user_id = 1 AND order_id = 1 |
+------------------+------------------------------------------------------------+
1 row in set (0.04 sec)

mysql> PREVIEW SELECT * FROM t_order_item WHERE user_id = 2 AND order_id = 5;
+------------------+-----------------------------------------------------------------+
| data_source_name | actual_sql                                                      |
+------------------+-----------------------------------------------------------------+
| ds_0             | SELECT * FROM t_order_item_2 WHERE user_id = 2 AND order_id = 5 |
+------------------+-----------------------------------------------------------------+
1 row in set (0.01 sec)

Single-table routes scan the fewest shard tables and offer the highest efficiency.

Query unused resources

During system maintenance, algorithms or storage resources that are no longer in use may need to be released, or resources that need to be released may have been referenced and cannot be deleted. DistSQL's SHOW UNUSED RESOURCES command can solve these problems:

mysql> ADD RESOURCE ds_2 (
    ->     HOST=127.0.0.1,
    ->     PORT=3306,
    ->     DB=demo_ds_2,
    ->     USER=root,
    ->     PASSWORD=123456
    -> );
Query OK, 0 rows affected (0.07 sec)

mysql> SHOW UNUSED RESOURCES\G;
*************************** 1. row ***************************
                           name: ds_2
                           type: MySQL
                           host: 127.0.0.1
                           port: 3306
                             db: demo_ds_2
connection_timeout_milliseconds: 30000
      idle_timeout_milliseconds: 60000
      max_lifetime_milliseconds: 2100000
                  max_pool_size: 50
                  min_pool_size: 1
                      read_only: false
               other_attributes: {"dataSourceProperties":{"cacheServerConfiguration":"true","elideSetAutoCommits":"true","useServerPrepStmts":"true","cachePrepStmts":"true","useSSL":"false","rewriteBatchedStatements":"true","cacheResultSetMetadata":"false","useLocalSessionState":"true","maintainTimeStats":"false","prepStmtCacheSize":"200000","tinyInt1isBit":"false","prepStmtCacheSqlLimit":"2048","serverTimezone":"UTC","netTimeoutForStreamingResults":"0","zeroDateTimeBehavior":"round"},"healthCheckProperties":{},"initializationFailTimeout":1,"validationTimeout":5000,"leakDetectionThreshold":0,"poolName":"HikariPool-8","registerMbeans":false,"allowPoolSuspension":false,"autoCommit":true,"isolateInternalQueries":false}
1 row in set (0.03 sec)Query unused primary key generator

DistSQL can also display unused sharding key generators with the SHOW UNUSED SHARDING KEY GENERATORS:

mysql> SHOW SHARDING KEY GENERATORS;
+--------------------------+-----------+-------+
| name                     | type      | props |
+--------------------------+-----------+-------+
| snowflake_key_generator  | snowflake | {}    |
| t_order_detail_snowflake | snowflake | {}    |
+--------------------------+-----------+-------+
2 rows in set (0.00 sec)

mysql> SHOW UNUSED SHARDING KEY GENERATORS;
Empty set (0.01 sec)

mysql> CREATE SHARDING KEY GENERATOR useless (
    -> TYPE(NAME=SNOWFLAKE)
    -> );
Query OK, 0 rows affected (0.04 sec)

mysql> SHOW UNUSED SHARDING KEY GENERATORS;
+---------+-----------+-------+
| name    | type      | props |
+---------+-----------+-------+
| useless | snowflake |       |
+---------+-----------+-------+
1 row in set (0.01 sec)Query unused sharding algorithm

DistSQL can reveal unused sharding algorithms with (you guessed it) the SHOW UNUSED SHARDING ALGORITHMS command:

mysql> SHOW SHARDING ALGORITHMS;
+--------------------------------+--------+-----------------------------------------------------+
| name                           | type   | props                                               |
+--------------------------------+--------+-----------------------------------------------------+
| database_inline                | inline | algorithm-expression=ds_${user_id % 2}              |
| t_order_inline                 | inline | algorithm-expression=t_order_${order_id % 3}        |
| t_order_item_inline            | inline | algorithm-expression=t_order_item_${order_id % 3}   |
| t_order_detail_database_inline | inline | algorithm-expression=ds_${user_id % 2}              |
| t_order_detail_table_inline    | inline | algorithm-expression=t_order_detail_${order_id % 3} |
+--------------------------------+--------+-----------------------------------------------------+
5 rows in set (0.00 sec)

mysql> CREATE SHARDING ALGORITHM useless (
    -> TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="ds_${user_id % 2}"))
    -> );
Query OK, 0 rows affected (0.04 sec)

mysql> SHOW UNUSED SHARDING ALGORITHMS;
+---------+--------+----------------------------------------+
| name    | type   | props                                  |
+---------+--------+----------------------------------------+
| useless | inline | algorithm-expression=ds_${user_id % 2} |
+---------+--------+----------------------------------------+
1 row in set (0.00 sec)Query rules that use the target storage resources

You can also see used resources within rules with SHOW RULES USED RESOURCE. All rules that use a resource can be queried, not limited to the sharding rule.

mysql> DROP RESOURCE ds_0;
ERROR 1101 (C1101): Resource [ds_0] is still used by [ShardingRule].

mysql> SHOW RULES USED RESOURCE ds_0;
+----------+----------------+
| type     | name           |
+----------+----------------+
| sharding | t_order        |
| sharding | t_order_item   |
| sharding | t_order_detail |
+----------+----------------+
3 rows in set (0.00 sec)Query sharding rules that use the target primary key generator

You can find sharding rules using a key generator with SHOW SHARDING TABLE RULES USED KEY GENERATOR:

mysql> SHOW SHARDING KEY GENERATORS;
+--------------------------+-----------+-------+
| name                     | type      | props |
+--------------------------+-----------+-------+
| snowflake_key_generator  | snowflake | {}    |
| t_order_detail_snowflake | snowflake | {}    |
| useless                  | snowflake | {}    |
+--------------------------+-----------+-------+
3 rows in set (0.00 sec)

mysql> DROP SHARDING KEY GENERATOR snowflake_key_generator;
ERROR 1121 (C1121): Sharding key generator `[snowflake_key_generator]` in database `sharding_db` are still in used.

mysql> SHOW SHARDING TABLE RULES USED KEY GENERATOR snowflake_key_generator;
+-------+--------------+
| type  | name         |
+-------+--------------+
| table | t_order      |
| table | t_order_item |
+-------+--------------+
2 rows in set (0.00 sec)Query sharding rules that use the target algorithm

Show sharding rules using a target algorithm with SHOW SHARDING TABLE RULES USED ALGORITHM:

mysql> SHOW SHARDING ALGORITHMS;
+--------------------------------+--------+-----------------------------------------------------+
| name                           | type   | props                                               |
+--------------------------------+--------+-----------------------------------------------------+
| database_inline                | inline | algorithm-expression=ds_${user_id % 2}              |
| t_order_inline                 | inline | algorithm-expression=t_order_${order_id % 3}        |
| t_order_item_inline            | inline | algorithm-expression=t_order_item_${order_id % 3}   |
| t_order_detail_database_inline | inline | algorithm-expression=ds_${user_id % 2}              |
| t_order_detail_table_inline    | inline | algorithm-expression=t_order_detail_${order_id % 3} |
| useless                        | inline | algorithm-expression=ds_${user_id % 2}              |
+--------------------------------+--------+-----------------------------------------------------+
6 rows in set (0.00 sec)

mysql> DROP SHARDING ALGORITHM t_order_detail_table_inline;
ERROR 1116 (C1116): Sharding algorithms `[t_order_detail_table_inline]` in database `sharding_db` are still in used.

mysql> SHOW SHARDING TABLE RULES USED ALGORITHM t_order_detail_table_inline;
+-------+----------------+
| type  | name           |
+-------+----------------+
| table | t_order_detail |
+-------+----------------+
1 row in set (0.00 sec)Make sharding better

DistSQL provides a flexible syntax to help simplify operations. In addition to the INLINE algorithm, DistSQL supports standard sharding, compound sharding, HINT sharding, and custom sharding algorithms.

If you have any questions or suggestions about Apache ShardingSphere, please feel free to post them on ShardingSphereGitHub.

Take a look at a data sharding scenario in which DistSQL's flexibility allows you to create a distributed database.

Image by:

Opensource.com

Databases What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Install JDBC on Linux in 3 steps

Fri, 09/23/2022 - 15:00
Install JDBC on Linux in 3 steps Seth Kenlon Fri, 09/23/2022 - 03:00

When you write an application, it's common to require data storage. Sometimes you're storing assets your application needs to function, and other times you're storing user data, including preferences and save data. One way to store data is in a database, and in order to communicate between your code and a database, you need a database binding or connector for your language. For Java, a common database connector is JDBC (Java database connectivity.)

1. Install Java

Of course, to develop with Java you must also have Java installed. I recommend SDKman for Linux, macOS, and WSL or Cygwin. For Windows, you can download OpenJDK from developers.redhat.com.

More on Java What is enterprise Java programming? Red Hat build of OpenJDK Java cheat sheet Free online course: Developing cloud-native applications with microservices Fresh Java articles 2. Install JDBC with Maven

JDBC is an API, imported into your code with the statement import java.sql.*, but for it to be useful you must have a database driver and a database installed for it to interact with. The database driver you use and the database you want to communicate with must match: to interact with MySQL, you need a MySQL driver, to interact with SQLite3, you must have the SQLite3 driver, and so on.

For this article, I use PostgreSQL, but all the major databases, including MariaDB and SQLite3, have JDBC drivers.

You can download JDBC for PostgreSQL from jdbc.postgresql.org. I use Maven to manage Java dependencies, so I include it in pom.xml (adjusting the version number for what's current on Maven Central):

>
    >org.postgresql>
    >postgresql>
    >42.5.0>
>3. Install the database

You have to install the database you want to connect to through JDBC. There are several very good open source databases, but I had to choose one for this article, so I chose PostgreSQL.

To install PostgreSQL on Linux, use your software repository. On Fedora, CentOS, Mageia, and similar:

$ sudo dnf install postgresql postgresql-server

On Debian, Linux Mint, Elementary, and similar:

$ sudo apt install postgresql postgresql-contribDatabase connectivity

If you're not using PostgreSQL, the same general process applies:

  1. Install Java.

  2. Find the JDBC driver for your database of choice and include it in your pom.xml file.

  3. Install the database (server and client) on your development OS.

Three steps and you're ready to start writing code.

Install Java, install JDBC with Maven, and install the database. Then you're ready to interact with databases in your Java code.

Image by:

Pixabay. CC0.

Java Linux Databases What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

5 Git configurations I make on Linux

Thu, 09/22/2022 - 15:00
5 Git configurations I make on Linux Alan Formy-Duval Thu, 09/22/2022 - 03:00

Setting up Git on Linux is simple, but here are the five things I do to get the perfect configuration:

  1. Create global configuration
  2. Set default name
  3. Set default email address
  4. Set default branch name
  5. Set default editor

I manage my code, shell scripts, and documentation versioning using Git. This means that for each new project I start, the first step is to create a directory for its content and make it into a Git repository:

$ mkdir newproject
$ cd newproject
$ git init

More on Git What is Git? Git cheat sheet Markdown cheat sheet New Git articles

There are certain general settings that I always want. Not many, but enough that I don't want to have to repeat the configuration each time. I like to take advantage of the global configuration capability of Git.

Git offers the git config command for manual configuration but this is a lot of work with certain caveats. For example, a common item to set is your email address. You can set it by running git config user.email followed by your email address. However, this will only take effect if you are in an existing Git directory:

$ git config user.email alan@opensource.com
fatal: not in a git directory

Plus, when this command is run within a Git repository, it only configures that specific one. The process must be repeated for new repositories. I can avoid this repetition by setting it globally. The --global option will instruct Git to write the email address to the global configuration file; ~/.gitconfig, even creating it if necessary:

  Remember, the tilde (~) character represents your home directory. In my case that is /home/alan. $ git config --global user.email alan@opensource.com
$ cat ~/.gitconfig
[user]
        email = alan@opensource.com

The downside here is if you have a large list of preferred settings, you will have a lot of commands to enter. This is time-consuming and prone to human error. Git provides an even more efficient and convenient way to directly edit your global configuration file—that is the first item on my list!

1. Create global configuration

If you have just started using Git, you may not have this file at all. Not to worry, let's skip the searching and get started. Just use the --edit option:

$ git config --global --edit

If no file is found, Git will generate one with the following content and open it in your shell environment's default editor:

# This is Git's per-user configuration file.
[user]
# Please adapt and uncomment the following lines:
#       name = Alan
#       email = alan@hopper
~
~
~
"~/.gitconfig" 5L, 155B                                     1,1           All

Now that we have opened the editor and Git has created the global configuration file behind the scenes, we can continue with the rest of the settings.

2. Set default name

Name is the first directive in the file, so let's start with that. The command line to set mine is git config --global user.name "Alan Formy-Duval". Instead of running this command, just edit the name directive in the configuration file:

name = Alan Formy-Duval3. Set default email address

The email address is the second directive, so let's update it. By default, Git uses your system-provided name and email address. If this is incorrect or you prefer something different, you can specify it in the configuration file. In fact, if you have not configured them, Git will let you know with a friendly message the first time you commit:

Committer: Alan <alan@hopper>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate....

The command line to set mine is git config --global user.email "alan@opensource.com". Instead, edit the email directive in the configuration file and provide your preferred address:

email = alan@opensource.com

The last two settings that I like to set are the default branch name and the default editor. These directives will need to be added while you are still in the editor.

4. Set default branch name

There is currently a trend to move away from the usage of the word master as the default branch name. As a matter of fact, Git will highlight this trend with a friendly message upon initialization of a new repository:

$ git init
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint:   git config --global init.defaultBranch <name>

This directive, named defaultBranch, needs to be located in a new section named init. It is now generally accepted that many coders use the word main for their default branch. This is what I like to use. Add this section followed by the directive to the configuration:

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles [init]
            defaultBranch = main5. Set default editor

The fifth setting that I like to set is the default editor. This refers to the editor that Git will present for typing your commit message each time you commit changes to your repository. Everyone has a preference whether it is nano, emacs, vi, or something else. I'm happy with vi. So, to set your editor, add a core section that includes the editor directive:

[core]
            editor = vi

That's the last one. Exit the editor. Git saves the global configuration file in your home directory. If you run the editing command again, you will see all of the content. Notice that the configuration file is a plain text file, so it can also be viewed using text tools such as the cat command. This is how mine appears:

$ cat ~/.gitconfig
[user]
        email = alan@opensource.com
        name = Alan Formy-Duval
[core]
        editor = vi
[init]
        defaultBranch = main

This is a simple guide to quickly get started working with Git and a few of its many configuration options. There are many other articles on Git here at Opensource.com, as well as our downloadable Git cheat sheet.

This is a simple guide to quickly get started working with Git and a few of its many configuration options.

Linux Git What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

6 Python interpreters to try in 2022

Wed, 09/21/2022 - 15:00
6 Python interpreters to try in 2022 Stephan Avenwedde Wed, 09/21/2022 - 03:00

Python, one of the most popular programming languages, requires an interpreter to execute the instructions defined by the Python code. In contrast to other languages, which compile directly into machine code, it’s up to the interpreter to read Python code and translate its instructions for the CPU performing the related actions. There are several interpreters out there, and in this article, I’ll take a look at a few of them.

Primer to interpreters

When talking about the Python interpreter, it’s usually the /usr/bin/python binary being referred to. That lets you execute a .py file.
However, interpreting is just one task. Before a line of Python code is actually executed on the CPU, these four steps are involved:

  1. Lexing - The human-made source code is converted into a sequence of logical entities, the so called lexical tokens.
  2. Parsing - In the parser, the lexical tokens are checked in regards of syntax and grammar. The output of the parser is an abstract syntax tree (AST).
  3. Compiling - Based on the AST, the compiler creates Python bytecode. The bytecode consists of very basic, platform independent instructions.
  4. Interpreting - The interpreter takes the bytecode and performs the specified operations.

As you can see, a lot of steps are required before any real action is taken. It makes sense to take a closer look at the different interpreters.

1. CPython

CPython is the reference implementation of Python and the default on many systems. As the name suggests, CPython is written in C.
As a result, it is possible to write extensions in C and therefore make the widley used C based library code available to Python. CPython is available on a wide range of platforms including ARM, iOS, and RISC. However, as the reference implementation of the language, CPython is carefully optimized and not focused on speed.

2. Pyston

Pyston is a fork of the CPython interpreter which implements performance optimizations. The project describes itself as a replacement of the standard CPython interpreter for large, real-world applications with a speedup potential up to 30%. Due to the lack of compatible binary packages, Pyston packages must be recompiled during the download process.

3. PyPy

PyPy is a Just-in-time (JIT) compiler for Python which is written in RPython, a statically typed subset of Python. In contrast to the CPython interpreter, PyPy compiles to machine code which can be directly executed by the CPU. PyPy is the playground for Python developers where they can experiment with new features more easily.

PyPy is faster than the reference CPython implementation. Because of the nature of JIT compiler, only applications that have been running for a long time benefit from caching.  PyPy can act as a replacement for CPython. There is a drawback, though. C-extension modules are mostly supported, but they run slower than a Python one. PyPy extension modules are written in Python (not C) and so the JIT compiler is able to optimized them. As long as your application isn't dependent on incompatible modules, PyPy is a great replacement for CPython. There is a dedicated page on the project website which describes the differences to CPython in detail: Diffrences between PyPy and CPython

4. RustPython

As the name suggest, RustPython is a Python interpreter written in Rust. Although the Rust programming language is quite new, it has been gaining popularity and is a candidate to be a successor of C and C++. By default, RustPython behaves like the interpreter of CPython but it also has a JIT compiler which can be enabled optionally. Another nice feature is that the Rust toolchain allows you to directly compile to WebAssembly and also allows you to run the interpreter completely in the browser. A demo of it can be found at rustpython.github.com/demo.

5. Stackless Python

Stackless Python describes itself as an enhanced version of the Python programming language. The project is basically a fork of the CPython interpreter which adds microthreads, channels and a scheduler to the language. Microthreads allow you to structure your code into tasklets which let you run your code in parallel. This approach is comparable to using green threads of the greenlet module. Channels can be used for bidirectional communication between tasklets. A famous user of Stackless Python is the MMORPG Eve Online.

More Python resources What is an IDE? Cheat sheet: Python 3.7 for beginners Top Python GUI frameworks Download: 7 essential PyPI libraries Red Hat Developers Latest Python articles 6. Micro Python

MicroPython is the way to go if you target micro controllers. It is a lean implementation that only requires 16kB of RAM and 256kB of space. Due to the embedded environment which it is intended for, MicroPython’s standard library is only a subset of CPython’s extensive STL. For developing and testing or as a lightweight alternative, MicroPython also runs on ordinary x86 and x64 machines. MicroPython is available for Linux, Windows, as well as many microcontrollers.

Performance

By design, Python is an inherently slow language. Depending on the task, there are significant performance differences between the interpreters. To get an overview of which interpreter is the best pick for a certain task, refer to pybenchmarks.org. An alternative to using an interpreter is to compile Python binary code directly into machine code. Nuitka, for example, is one of those projects which can compile Python code to C code and from C to machine code. The C code is then compiled to machine code using an ordinary C compiler. The topic of Python compilers is quite comprehensive and worth a separate article.

Summary

Python is a wonderful language for rapid prototyping and automating tasks. Additionally, it is easy to learn and well suited for beginners. If you usually stick with CPython, it could be interesting to see how your code behaves on another interpreter. If you use Fedora, you can easily test a few other interpreters as the package manager already provides the right binaries. Check out fedora.developer.org for more information.

It could be interesting to see how your code behaves on another interpreter than what you're used to.

Image by:

WOCinTech Chat. Modified by Opensource.com. CC BY-SA 4.0

Python Programming What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

My favorite open source alternatives to Notion

Wed, 09/21/2022 - 15:00
My favorite open source alternatives to Notion Amar Gandhi Wed, 09/21/2022 - 03:00

If you have notes to yourself scattered throughout your hard drive, you might need a notes application to collect and organize your personal reminders. A notes system can help you track ideas, important tasks, and works in progress. A popular application that isn't open source is Notion, but here are two options that respect your privacy and data.

Standard Notes

Standard Notes is an open source (AGPL 3.0 license) notes application featuring a password manager, a to-do list, and, of course, a great system for writing and storing notes.

One of the most important things about taking notes is finding them again, so organization is critical. Standard Notes uses an intuitive and natural tagging system to help you organize your content. You assign hashtags to each note to classify it.

Standard Notes is extensible through plug-ins. There are plug-ins for LaTeX, Markdown, code snippets, spreadsheets, and more. There's even an option to publish to a blogging platform, should you want to make some of your notes public.

Image by:

(Amir Gandhi, CC BY-SA 4.0)

Standard Notes also boasts numerous backup options, including email and cloud services. Furthermore, Standard Notes can work on any platform, including Linux, Windows, macOS, and Chrome OS.

More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources Self-hosting Standard Notes

Standard Notes can be self-hosted. The developers provide a script that runs the application in a container, making it to run almost anywhere. If you've yet to explore containers, then you can get up to speed with Opensource.com's introduction to running applications in containers.

Another option is to use the hosted version provided by Standard Notes.

The development of Standard Notes can be followed on its Git repository.

Trilium

Trilium is a notes application that visually resembles Notion in many ways. It can handle various data types, including images, tables, to-do lists, highlighting, mind maps, flowcharts, family trees, code blocks, and more.

Trilium has several mechanisms to help you organize both your thoughts and your notes. You can view a history of recent changes, a global map of all your notes, note categories, or you can search for notes and contents.

Image by:

(Amir Gandhi, CC BY-SA 4.0)

You can install Trilium as a Flatpak from Flathub, or you can install it on your own server as a container. Alternatively, you can use Trilium's hosted instance.

Take note

There are plenty of useful note-taking applications in the open source world, and both Standard Notes and Trilium are designed with your data as the top priority. You can import and export data from these applications, so it's safe to try them out. You'll always have access to your data, so give Standard Notes or Trilium a try.

There are lots of useful open source note-taking tools out there. Standard Notes and Trilium are designed with your data as the top priority.

Alternatives What to read next 5 note-taking apps for Linux This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

3 ways to use the Linux inxi command

Tue, 09/20/2022 - 15:00
3 ways to use the Linux inxi command Don Watkins Tue, 09/20/2022 - 03:00

I was looking for information about the health of my laptop battery when I stumbled upon inxi. It's a command line system information tool that provides a wealth of information about your Linux computer, whether it's a laptop, desktop, or server.

The inxi command is licensed with the GPLv3, and many Linux distributions include it. According to its Git repository: "inxi strives to support the widest range of operating systems and hardware, from the most simple consumer desktops, to the most advanced professional hardware and servers."

Documentation is robust, and the project maintains a complete man page online. Once installed, you can access the man page on your system with the man inxi command.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Install inxi on Linux

Generally, you can install inxi from your distribution's software repository or app center. For example, on Fedora, CentOS, Mageia, or similar:

$ sudo dnf install inxi

On Debian, Elementary, Linux Mint, or similar:

$ sudo apt install inxi

You can find more information about installation options for your Linux distribution here.

3 ways to use inxi on Linux

Once you install inxi, you can explore all its options. There are numerous options to help you learn more about your system. The most fundamental command provides a basic overview of your system:

$ inxi -b
System:
  Host: pop-os Kernel: 5.19.0-76051900-generic x86_64 bits: 64
        Desktop: GNOME 42.3.1 Distro: Pop!_OS 22.04 LTS
Machine:
  Type: Laptop System: HP product: Dev One Notebook PC v: N/A
        serial: <superuser required>
  Mobo: HP model: 8A78 v: KBC Version 01.03 serial: <superuser required>
        UEFI: Insyde v: F.05 date: 06/14/2022
Battery:
  ID-1: BATT charge: 50.6 Wh (96.9%) condition: 52.2/53.2 Wh (98.0%)
CPU:
  Info: 8-core AMD Ryzen 7 PRO 5850U with Radeon Graphics [MT MCP]
        speed (MHz): avg: 915 min/max: 400/4507
Graphics:
  Device-1: AMD Cezanne driver: amdgpu v: kernel
  Device-2: Quanta HP HD Camera type: USB driver: uvcvideo
  Display: x11 server: X.Org v: 1.21.1.3 driver: X: loaded: amdgpu,ati
        unloaded: fbdev,modesetting,radeon,vesa gpu: amdgpu
        resolution: 1920x1080~60Hz
  OpenGL:
        renderer: AMD RENOIR (LLVM 13.0.1 DRM 3.47 5.19.0-76051900-generic)
        v: 4.6 Mesa 22.0.5
Network:
  Device-1: Realtek RTL8822CE 802.11ac PCIe Wireless Network Adapter
        driver: rtw_8822ce
Drives:
  Local Storage: total: 953.87 GiB used: 75.44 GiB (7.9%)
Info:
  Processes: 347 Uptime: 15m Memory: 14.96 GiB used: 2.91 GiB (19.4%)
  Shell: Bash inxi: 3.3.131. Display battery status

You can check your battery health using the -B option. The result shows the system battery ID, charge condition, and other information:

$ inxi -B
Battery:
ID-1: BATT charge: 44.3 Wh (85.2%) condition: 52.0/53.2 Wh (97.7%)2. Display CPU info

Find out more information about the CPU with the -C option:

$ inxi -C CPU: Info: 8-core model: AMD Ryzen 7 PRO 5850U with Radeon Graphics bits: 64 type: MT MCP cache: L2: 4 MiB Speed (MHz): avg: 400 min/max: 400/4507 cores: 1: 400 2: 400 3: 400 4: 400 5: 400 6: 400 7: 400 8: 400 9: 400 10: 400 11: 400 12: 400 13: 400 14: 400 15: 400 16: 400

The output of inxi uses colored text by default. You can change that to improve readability, as needed, by using the "color switch."

The command option is -c followed by any number between 0 and 42 to suit your tastes.

$ inxi -c 42

Here is an example of a couple of different options using color 5 and then 7:

Image by:

(Don Watkins, CC BY-SA 4.0)

The software can show hardware temperature, fan speed, and other information about your system using the sensors in your Linux system. Enter inxi -s and read the result below:

Image by:

(Don Watkins, CC BY-SA 4.0)

3. Combine options

You can combine options for inxi to get complex output when supported. For example, inxi -S provides system information, and -v provides verbose output. Combining the two gives the following:

$ inxi -S
System:
  Host: pop-os Kernel: 5.19.0-76051900-generic x86_64 bits: 64
        Desktop: GNOME 42.3.1 Distro: Pop!_OS 22.04 LTS

$ inxi -Sv
CPU: 8-core AMD Ryzen 7 PRO 5850U with Radeon Graphics (-MT MCP-)
speed/min/max: 634/400/4507 MHz Kernel: 5.19.0-76051900-generic x86_64
Up: 20m Mem: 3084.2/15318.5 MiB (20.1%) Storage: 953.87 GiB (7.9% used)
Procs: 346 Shell: Bash inxi: 3.3.13Bonus: Check the weather

Your computer isn't all inxi can gather information about. With the -w option, you can also get weather information for your locale:

$ inxi -w
Weather:
  Report: temperature: 14 C (57 F) conditions: Clear sky
  Locale: Wellington, G2, NZL
        current time: Tue 30 Aug 2022 16:28:14 (Pacific/Auckland)
        Source: WeatherBit.io

You can get weather information for other areas of the world by specifying the city and country you want along with -W:

$ inxi -W rome,italy
Weather:
  Report: temperature: 20 C (68 F) conditions: Clear sky
  Locale: Rome, Italy current time: Tue 30 Aug 2022 06:29:52
        Source: WeatherBit.ioWrap up

There are many great tools to gather information about your computer. I use different ones depending on the machine, the desktop, or my mood. What are your favorite system information tools?

I use inxi on Linux to check my laptop batter, CPU information, and even the weather.

Linux Command line Hardware What to read next Linux commands to display your hardware information This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Security buzzwords to avoid and what to say instead

Tue, 09/20/2022 - 15:00
Security buzzwords to avoid and what to say instead Seth Kenlon Tue, 09/20/2022 - 03:00

Technology is a little famous for coming up with "buzzwords." Other industries do it, too, of course. "Story-driven" and "rules light" tabletop games are a big thing right now, "deconstructed" burgers and burritos are a big deal in fine dining. The problem with buzzwords in tech, though, is that they potentially actually affect your life. When somebody calls an application "secure," to influence you to use their product, there's an implicit promise being made. "Secure" must mean that something's secure. It's safe for you to use and trust. The problem is, the word "secure" can actually refer to any number of things, and the tech industry often uses it as such a general term that it becomes meaningless.

Because "secure" can mean both so much and so little, it's important to use the word "secure" carefully. In fact, it's often best not to use the word at all, and instead, just say what you actually mean.

More on security The defensive coding guide 10 layers of Linux container security SELinux coloring book More security articles When "secure" means encrypted

Sometimes "secure" is imprecise shorthand for encrypted. In this context, "secure" refers to some degree of difficulty for outside observers to eavesdrop on your data.

Don't say this: "This website is resilient and secure."

That sounds pretty good! You're probably imagining a website that has several options for 2-factor authentication, zero-knowledge data storage, and steadfast anonymity policies.

Say this instead: "This website has a 99% uptime guarantee, and its traffic is encrypted and verifiable with SSL."

Not only is the intent of the promise clear now, it also explains how "secure" is achieved (it uses SSL) and what the scope of "secure" is.

Note that there's explicitly no promise here about privacy or anonymity.

When "secure" means restricted access

Sometimes the term "secure" refers to application or device access. Without clarification, "secure" could mean anything from the useless security by obscurity model, to a simple htaccess password, to biometric scanners.

Don't say this: "We've secured the system for your protection."

Say this instead: "Our system uses 2-factor authentication."

When "secure" means data storage

The term "secure" can also refer to the way your data is stored on a server or a device.

Don't say this: "This device stores your data with security in mind."

Say this instead: "This device uses full disk encryption to protect your data."

When remote storage is involved, "secure" may instead refer to who has access to stored data.

Don't say this: "Your data is secure."

Say this instead: "Your data is encrypted using PGP, and only you have the private key."

When "secure" means privacy

These days, the term "privacy" is almost as broad and imprecise as "security." On one hand, you might think that "secure" must mean "private," but that's true only when "secure" has been defined. Is something private because it has a password barrier to entry? Or is something private because it's been encrypted and only you have the keys? Or is it private because the vendor storing your data knows nothing about you (aside from an IP address?) It's not enough to declare "privacy" any more than it is to declare "security" without qualification.

Don't say this: "Your data is secure with us."

Say this instead: "Your data is encrypted with PGP, and only you have the private key. We require no personal data from you, and can only identify you by your IP address."

Some sites make claims about how long IP addresses are retained in logs, and promises about never surrendering data to authorities without warrants, and so on. Those are beyond the scope of technological "security," and have everything to do with trust, so don't confuse them for technical specifications.

Say what you mean

Technology is a complex topic with a lot of potential for confusion. Communication is important, and while shorthand and jargon can be useful in some settings, generally it's better to be precise. When you're proud of the "security" of your project, don't generalize it with a broad term. Make it clear to others what you're doing to protect your users, and make it equally clear what you consider out of scope, and communicate these things often. "Security" is a great feature, but it's a broad one, so don't be afraid to brag about the specifics.

Consider these thoughtful approaches to define what security really means in your open source project.

Image by:

JanBaby, via Pixabay CC0.

Security and privacy What to read next Why transparency is critical to your open source project's security Encrypting and decrypting files with OpenSSL This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

I got my first pull request merged!

Mon, 09/19/2022 - 15:00
I got my first pull request merged! Oluwaseun Mon, 09/19/2022 - 03:00

Words cannot express my joy when I got the notification about the merge below, and I owe it to my current engineering school, AltSchool Africa.

Image by:

(Awosise Oluwaseun, CC BY-SA 4.0)

Before this, I had been introduced to open source many times, told about its importance in the tech space, and even attended open source conferences (e.g., OSCAFest). I had all the instant excitement to start, but imposter syndrome set in on opening GitHub to create something.

Fast forward to Monday, the 8th of August, 2022, when I watched Bolaji's video on contributing to open source. I felt pumped again, but I wanted to apply what I learned, so I noted some steps.

The steps:

  1. I made up my mind I was going to contribute to a project.
  2. I was focused on a site (good first issue) to pick my first project from, which I filtered to suit my skill level. I kept opening the next page till I found one.
  3. I made sure I was equipped with the required Git and GitHub knowledge to complete the project.

More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources The project

After long hours searching for projects, I finally found one titled, Ensure no missing alt attributes. I was to give descriptive alt values to images from the site. Alt values in images help to improve the accessibility of the site such that screen readers can provide a detailed description of the image to, say, a visually impaired person. Easy right? Yes, but if I didn't make up my mind to get the first contribution, I wouldn't find it, and open source would continue to be a myth to me.

I was still pumped until I discovered it was from MDN. Wait, MDN? As in Mozilla developer? Will they merge my contribution even with how seemingly easy it looks? Imposter syndrome set in.

Upon checking the issue, I saw that people were already contributing. I summoned my courage and started reading about it. Taking my time to read and understand the project and how I needed to approach the issue was another challenge I had to overcome.

The project is as easy as you try to understand it.

So, I picked two images to begin with. I gave alt values to them, committed my changes, then made a pull request. The time between when I made the pull request and when I got the approval mail was full of self-doubts. Should I close the pull request? This is MDN. Well, it's not coding... What if I don't get merged? I might never contribute again. All it took to clear all of the doubts were the emails I got from my reviewer below:

Image by:

(Awosise Oluwaseun, CC BY-SA 4.0)

Image by:

(Awosise Oluwaseun, CC BY-SA 4.0)

Image by:

(Awosise Oluwaseun, CC BY-SA 4.0)

I was indeed delighted, and this inspired me to check for more. It gave me the courage I needed to request additional issues to solve.

Image by:

(Awosise Oluwaseun, CC BY-SA 4.0)

Summary

A few lessons I'd love you to take home from this article are:

  • Open source is for all. Do you see that typo on that site you just visited? You helping to correct it is a way of contributing.
  • No skillset is too small. A basic understanding of HTML was what I needed to contribute.
  • Only you can stop yourself from contributing.
  • The first contribution is all you need to get the ball rolling.

I hope you have been able to pick something from my story and apply it today. This is another space I'd like to keep contributing to, so see you in my next article, and happy open sourcing!

This article originally appeared on I got my first Pull Request merged! and is republished with permission.

Experience the joy that contributing to open source brings.

Image by:

Photo by Rob Tiller, CC BY-SA 4.0

Community management What to read next My first contribution to open source: Making a decision New open source tool catalogs African language resources This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

PyLint: The good, the bad, and the ugly

Mon, 09/19/2022 - 15:00
PyLint: The good, the bad, and the ugly Moshe Zadka Mon, 09/19/2022 - 03:00

Hot take: PyLint is actually good!

"PyLint can save your life" is an exaggeration, but not as much as you might think! PyLint can keep you from really really hard to find and complicated bugs. At worst, it can save you the time of a test run. At best, it can help you avoid complicated production mistakes.

The good

I'm embarrassed to say how common this can be. Naming tests is perpetually weird: Nothing cares about the name, and there's often not a natural name to be found. For instance, look at this code:

def test_add_small():
    # Math, am I right?
    assert 1 + 1 == 3
   
def test_add_large():
    assert 5 + 6 == 11
   
def test_add_small():
    assert 1 + 10 == 11

The test works:

collected 2 items                                                                        
test.py ..
2 passed

More Python resources What is an IDE? Cheat sheet: Python 3.7 for beginners Top Python GUI frameworks Download: 7 essential PyPI libraries Red Hat Developers Latest Python articles But here's the kicker: If you override a name, the testing infrastructure happily skips over the test!

In reality, these files can be hundreds of lines long, and the person adding the new test might not be aware of all the names. Unless someone is looking at test output carefully, everything looks fine.

Worst of all, the addition of the overriding test, the breakage of the overridden test, and the problem that results in prod might be separated by days, months, or even years.

PyLint finds it

But like a good friend, PyLint is there for you.

test.py:8:0: E0102: function already defined line 1
     (function-redefined)The bad

Like a 90s sitcom, the more you get into PyLint, the more it becomes problematic. This is completely reasonable code for an inventory modeling program:

"""Inventory abstractions"""

import attrs

@attrs.define
class Laptop:
    """A laptop"""
    ident: str
    cpu: str

It seems that PyLint has opinions (probably formed in the 90s) and is not afraid to state them as facts:

$ pylint laptop.py | sed -n '/^laptop/s/[^ ]*: //p'
R0903: Too few public methods (0/2) (too-few-public-methods)The ugly

Ever wanted to add your own unvetted opinion to a tool used by millions? PyLint has 12 million monthly downloads.

"People will just disable the whole check if it's too picky." —PyLint issue 6987, July 3rd, 2022

The attitude it takes towards adding a test with potentially many false positives is..."eh."

Making it work for you

PyLint is fine, but you need to interact with it carefully. Here are the three things I recommend to make PyLint work for you.

1. Pin it

Pin the PyLint version you use to avoid any surprises!

In your .toml file:

[project.optional-dependencies]
pylint = ["pylint"]

In your code:

from unittest import mock

This corresponds with code like this:

# noxfile.py
...
@nox.session(python=VERSIONS[-1])
def refresh_deps(session):
    """Refresh the requirements-*.txt files"""
    session.install("pip-tools")
    for deps in [..., "pylint"]:
        session.run(
            "pip-compile",
            "--extra",
            deps,
            "pyproject.toml",
            "--output-file",
            f"requirements-{deps}.txt",
        )2. Default deny

Disable all checks. Then enable ones that you think have a high value-to-false-positive ratio. (Not just false-negative-to-false-positive ratio!)

# noxfile.py
...
@nox.session(python="3.10")
def lint(session):
    files = ["src/", "noxfile.py"]
    session.install("-r", "requirements-pylint.txt")
    session.install("-e", ".")
    session.run(
        "pylint",
        "--disable=all",
        *(f"--enable={checker}" for checker in checkers)
        "src",
    )3. Checkers

These are some of the ones I like. Enforce consistency in the project, avoid some obvious mistakes.

checkers = [
    "missing-class-docstring",
    "missing-function-docstring",
    "missing-module-docstring",
    "function-redefined",
]Using PyLint

You can take just the good parts of PyLint. Run it in CI to keep consistency, and use the highest value checkers.

Lose the bad parts: Default deny checkers.

Avoid the ugly parts: Pin the version to avoid surprises.

Get the most out of PyLint.

Image by:

Opensource.com

Python What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

What is OpenRAN?

Sat, 09/17/2022 - 15:00
What is OpenRAN? Stephan Avenwedde Sat, 09/17/2022 - 03:00

If you own and use a smartphone capable of connecting to arbitrary computers all over the world, then you are a user of Radio Access Networks (RAN). A RAN is provided by your cellular provider, and it handles wireless connections between your smartphone and your communication provider.

While your smartphone may be running an open source operating system (Android) and the server you try to access is probably running Linux, there's a lot of proprietary technology in between to make the connection happen. While you may have a basic understanding of how networking works locally, this knowledge stops when you plug a SIM card into your smartphone in order to make a connection with a cell tower possible. In fact, the majority of software and hardware components in and around a cell tower are still closed source, which of course has some drawbacks. This is where OpenRAN comes into play.

The OpenRAN initiative (shorthand for Open Radio Access Network) was started by the O-Ran Alliance, a worldwide community of mobile operators, vendors, and research and academic institutions. The initiative aims to define open standards between the various components of radio access networks. Interoperability between components of different manufacturers was not possible. Until now.

More on edge computing Understanding edge computing Why Linux is critical to edge computing eBook: Running Kubernetes on your Raspberry Pi Download now: The automated enterprise eBook eBook: A practical guide to home automation using open source tools eBook: 7 examples of automation on the edge The latest on edge Radio Access Network

But what exactly is a RAN? In a nutshell, a RAN establishes a wireless connection to devices (smartphones, for example) and connects them to the core network of the communication company. In the context of a RAN, devices are denoted as User Equipment (UE).

The tasks of a RAN can be summarized as follows:

  • Authentication of UE
  • The handover of UE to another RAN (if the UE is moving)
  • Forwarding the data between the UE and the core network
  • Provision of the data for accounting functions (billing of services or the transmitted data)
  • Control of access to the various services
OpenRAN

RAN usually consists of proprietary components. OpenRAN defines functional units and interfaces between them:

  • Radio Unit (RU): The RU is connected to the antenna and sends, receives, amplifies, and digitizes radio signals.
  • Distributed Unit (DU): Handles the PHY, MAC and RLC layer.
  • Centralised Unit (CU): Handles the RRC and PDCP layer.
  • RAN Intelligent Controller (RIC): Control and optimization of RAN elements and resources.

Units are connected to each other by standardized, open interfaces. Furthermore, if the units can be virtualized and deployed in the cloud or on an edge device, then it's called a vRAN (virtual Radio Access Network). The basic principle of vRAN is to decouple the hardware from the software by using a software-based virtualization layer. Using a vRAN improves flexibility in terms of scalability and the underlying hardware.

OpenRAN for everyone

By the definition of functional units and the interfaces between them, OpenRAN enables interoperability of components from different manufacturers. This reduces the dependency for cellular providers of specific vendors and makes communication infrastructure more flexible and resilient. As a side-effect, using clearly defined functional units and interfaces drives innovation and competition. With vRAN, the use of standard hardware is possible. With all these advantages, OpenRAN is a prime example of how open source benefits everyone.

Open Radio Access Network defines open standards between the various components of radio access networks.

Image by:

Opensource.com

Edge computing What to read next Streaming internet radio with RadioDroid PyRadio: An open source alternative for internet radio This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Fix the apt-key deprecation error in Linux

Fri, 09/16/2022 - 15:00
Fix the apt-key deprecation error in Linux Chris Hermansen Fri, 09/16/2022 - 03:00

This morning, after returning home from a mini vacation, I decided to run apt update and apt upgrade from the command line just to see whether there had been any updates while I was offline. After issuing the update command, something didn't seem quite right; I was seeing messages along the lines of:

W: https://updates.example.com/desktop/apt/dists/xenial/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

True, it's just a warning, but still there's that scary word, deprecation, which usually means it's going away soon. So I thought I should take a look. Based on what I found, I thought my experience would be worth sharing.

It turns out that I have older configurations for some repositories, artifacts of installation processes from "back in the day," that needed adjustment. Taking my prompt from the warning message, I ran man apt-key at the command line, which provided several interesting bits of information. Near the beginning of the man page:

apt-key is used to manage the list of keys used by apt to authenticate packages. Packages which have been authenticated using these keys are considered trusted.
Use of apt-key is deprecated, except for the use of apt-key del in maintainer scripts to remove existing keys from the main keyring. If such usage of apt-key is desired, the additional installation of the GNU Privacy Guard suite (packaged in gnupg) is required.
apt-key(8) will last be available in Debian 11 and Ubuntu 22.04.

Last available in "Debian 11 and Ubuntu 22.04" is pretty much right now for me. Time to fix this!

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Fixing the apt-key deprecation error

Further on in the man page, there's the deprecation section mentioned in the warning from apt update:

DEPRECATION
Except for using apt-key del in maintainer scripts, the use of apt-key is deprecated. This section shows how to replace the existing use of apt-key.
If your existing use of apt-key add looks like this:

wget -qO- https://myrepo.example/myrepo.asc | sudo apt-key add -

Then you can directly replace this with (though note the recommendation below):

wget -qO- https://myrepo.example/myrepo.asc | sudo tee /etc/apt/trusted.gpg.d/myrepo.asc

Make sure to use the "asc" extension for ASCII armored keys and the "gpg" extension for the binary OpenPGP format (also known as "GPG key public ring"). The binary OpenPGP format works for all apt versions, while the ASCII armored format works for apt version >= 1.4.

Recommended: Instead of placing keys into the /etc/apt/trusted.gpg.d directory, you can place them anywhere on your filesystem by using the Signed-By option in your sources.list and pointing to the filename of the key. See sources.list(5) for details. Since APT 2.4, /etc/apt/keyrings is provided as the recommended location for keys not managed by packages. When using a deb822-style sources.list, and with apt version >= 2.4, the Signed-By option can also be used to include the full ASCII armored keyring directly in the sources.list without an additional file.

If you, like me, have keys from non-repository stuff added with apt-key, then here are the steps to transition:

  1. Determine which keys are in apt-key keyring /etc/apt/trusted.gpg
  2. Remove them
  3. Find and install replacements in /etc/apt/trusted.gpg.d/ or in /etc/apt/keyrings/
1. Finding old keys

The command apt-key list shows the keys in /etc/apt/trusted.gpg:

$ sudo apt-key list
[sudo] password:
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
/etc/apt/trusted.gpg
--------------------
pub   rsa4096 2017-04-05 [SC]
      DBE4 6B52 81D0 C816 F630  E889 D980 A174 57F6 FB86
uid           [ unknown] Example <support@example.com>
sub   rsa4096 2017-04-05 [E]

pub   rsa4096 2016-04-12 [SC]
      EB4C 1BFD 4F04 2F6D DDCC  EC91 7721 F63B D38B 4796
uid           [ unknown] Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>
sub   rsa4096 2021-10-26 [S] [expires: 2024-10-25]
[...]

Also shown afterward are the keys held in files in the /etc/apt/trusted.gpg.d folder.

[ Related read How to import your existing SSH keys into your GPG key ]

2. Removing old keys

The group of quartets of hex digits, for example DBEA 6B52...FB86, is the identifier required to delete the unwanted keys:

$ sudo apt-key del "DBEA 6B52 81D0 C816 F630  E889 D980 A174 57F6 FB86"

This gets rid of the Example key. That's literally just an example, and in reality you'd get rid of keys that actually exist. For instance, I ran the same command for each of the real keys on my system, including keys for Google, Signal, and Ascensio. Keys on your system will vary, depending on what you have installed.

3. Adding keys

Getting the replacement keys is dependent on the application. For example, Open Whisper offers its key and an explanation of what to do to install it, which I decided not to follow as it puts the key in /usr/share/keyrings. Instead, I did this:

$ wget -O- https://updates.signal.org/desktop/apt/keys.asc | gpg --dearmor > signal-desktop-keyring.gpg
$ sudo mv signal-desktop-keyring.gpg /etc/apt/trusted.gpg.d/
$ sudo chown root:root /etc/apt/trusted.gpg.d/signal-desktop-keyring.gpg
$ sudo chmod ugo+r /etc/apt/trusted.gpg.d/signal-desktop-keyring.gpg
$ sudo chmod go-w /etc/apt/trusted.gpg.d/signal-desktop-keyring.gpg

Ascencio also offers instructions for installing OnlyOffice that include dealing with the GPG key. Again I modified their instructions to suit my needs:

$ gpg --no-default-keyring --keyring gnupg-ring:~/onlyoffice.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys CB2DE8E5
$ sudo mv onlyoffice.gpg /etc/apt/trusted.gpg.d/
$ sudo chown root:root /etc/apt/trusted.gpg.d/onlyoffice.gpg
$ sudo chmod ugo+r /etc/apt/trusted.gpg.d/onlyoffice.gpg
$ sudo chmod go-w /etc/apt/trusted.gpg.d/onlyoffice.gpg

As for the Google key, it is managed (correctly, it appears) through the .deb package, and so a simple reinstall with dpkg -i was all that was needed. Finally, I ended up with this:

$ ls -l /etc/apt/trusted.gpg.d
total 24
-rw-r--r-- 1 root root 7821 Sep  2 10:55 google-chrome.gpg
-rw-r--r-- 1 root root 2279 Sep  2 08:27 onlyoffice.gpg
-rw-r--r-- 1 root root 2223 Sep  2 08:02 signal-desktop-keyring.gpg
-rw-r--r-- 1 root root 2794 Mar 26  2021 ubuntu-keyring-2012-cdimage.gpg
-rw-r--r-- 1 root root 1733 Mar 26  2021 ubuntu-keyring-2018-archive.gpgExpired keys

The last problem key I had was from an outdated installation of QGIS. The key had expired, and I'd set it up to be managed by apt-key. I ended up following their instructions to the letter, both for installing a new key in /etc/apt/keryings and their suggested format for the /etc/apt/sources.list.d/qgis.sources installation configuration.

[ Download the Linux cheat sheets for apt or dnf ]

Linux system maintenance

Now you can run apt update with no warnings or errors related to deprecated key configurations. We apt users just need to remember to adjust any old installation instructions that depend on apt-key. Instead of using apt-key, you must instead install a key to /etc/apt/trusted.gpg.d/ or /etc/apt/keyrings/, using gpg as needed.

Follow these steps and you can run apt update with no warnings or errors related to deprecated key configurations.

Image by:

Opensource.com

Linux Sysadmin What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Pages