Open-source News

With the Apple M2 running Asahi Linux you may be wondering whether it's better to use the GCC compiler as is the default on upstream Arch Linux or whether going for LLVM Clang will yield better performance given all the LLVM/Clang usage by AArch64 vendors, including Apple's own Xcode compiler toolchain making use of it. If you are wondering about GCC vs. Clang for building binaries on the Apple M2, here are some benchmarks.

Microsoft to Use the Mariner Linux Distro Exclusively for Its Xbox Storefronts  WinBuzzer

Microsoft to Use the Mariner Linux Distro Exclusively for Its Xbox Storefronts  WinBuzzer

Linux devices 'increasingly' under attack from hackers, warn security researchers  ZDNet

Back in 2013 when AMD was pushing their Heterogeneous System Architecture (HSA) they joined The Document Foundation and wanted to make use of OpenCL acceleration within this open-source office suite. Shortly thereafter they added many OpenCL functions to LibreOffice but now a decade later it seems to be of little use but at least this week thanks to a Collabora engineer there has been some OpenCL code cleaning for this free software office suite...

Inspired by Microsoft's compiler toolchain having a "std:c++latest" option for automatically targeting the latest supported C++ spec, GCC compiler developers have been discussing the possibility of a similar feature with the ability to specify -std=c++current for the current C++ standard or -std=c++future for the future/draft specification...

One of the interesting capabilities with Thunderbolt and now USB4 that is seemingly not too widely used is for networking between systems. The Linux kernel for the past half-decade already has offered a Thunderbolt networking driver for networking between hosts with Thunderbolt cables. The latest improvement on this front is now supporting USB4's end-to-end flow control mode...

Linux 6.1 Thunderbolt Networking To Support USB4 End-To-End Flow Control  Phoronix

For the open-source software popular for screen-casting and videogame live-streaming, OBS Studio 28.0 was released today as a big feature update...

The post How to Install Rocky Linux 9.0 Step by Step first appeared on Tecmint: Linux Howtos, Tutorials & Guides .

Rocky Linux 9.0 was made generally available on July 14, 2022. It comes with a boatload of new features and improvements and will enjoy support and maintenance until May 31, 2032. The previous release,

The post How to Install Rocky Linux 9.0 Step by Step first appeared on Tecmint: Linux Howtos, Tutorials & Guides.

Usability and accessibility starts with open communication Klaatu Thu, 09/01/2022 - 03:00 1 reader likes this 1 reader likes this

Amazing though it may seem, we each experience the world differently. That's one reality with over 6 billion interpretations. Many of us use computers to broaden our experience of the world, but a computer is part of reality and so if you experience reality without, for instance, vision or sound, then you also experience a computer without vision or sound (or whatever your unique experience might be.) As humans, we don't quite have the power to experience the world the way somebody does. We can mimic some of the surface-level things (I can close my eyes to mimic blindness, for example) but it's only an imitation, without history, context, or urgency. As a result of this complexity, we humans design things primarily for ourselves, based on the way we experience the world. That can be frustrating, from an engineering and design viewpoint, because even when you intend to be inclusive, you end up forgetting something "obvious" and essential, or the solution to one problem introduces a problem for someone else, and so on. What's an open source enthusiast, or programmer, or architect, or teacher, or just everyday hacker, supposed to do to make software, communities, and processes accessible?

Don't miss the opportunities

A friend of mine, who lives with hearing loss, recently signed up for a webinar and contacted the host to request captioning or, failing that, a transcript of the lessons. It was a great disappointment when the host, who had specifically emailed all participants with an invitation for feedback, never even responded to the request. In the end, some mutual friends attended the webinar and took notes.

[ Also read My open source journey with C from a neurodiverse perspective ]

The webinar was a small event run by an individual, so it's possible that emails all around were going unanswered until the end of the multi-week event. However, this incident can serve as a valuable lesson: Accessibility starts with communication.

You can't know the unique needs of every single person interacting with the thing (website, software, podcast, article, and so on) you produce. You can't predict what small arbitrary choice you make might lead to the accidental exclusion of someone who would otherwise have engaged with you. What you can do, though, is look for opportunities to learn about them. When someone sends an email about how the 8-point, thin, 45% gray font on a white background makes your website hard to read, don't ignore it, and don't chalk it up to a difference in opinion. When someone files a bug that Orca or NVDA can't navigate your application, don't close it until it's fixed.

What to do when you can't help

Nobody knows everything, and that's true for each of us participating in open source. It's very likely that you'll get a comment from somebody with an issue in something you've designed, and you won't know how to fix it. Or you might know how to fix it, but you just won't have the time to implement the fix. That doesn't make you a bad person, it just reveals the one thing that's true for all of us: You have limited resources. But through open collaboration, there's more than likely an answer.

Open source is all about sharing, and this is as true for code as it is for community resources. Identifying a bug at the very least demonstrates what your project needs from potential future contributors. Possibly, the person making the request or filing the bug can help you find someone who knows how to fix the issue. Or maybe they have friends who help them find a work-around, and could at the very least document the round-about way they deal with the issue, which could be exactly the stop-gap you need while you upskill enough to find the "right" fix for the problem.

[ Related read A practical guide to light and dark mode in Jekyll ]

Answers to usability and accessibility aren't always as direct as you think they need to be. Sometimes, a simple work-around or accommodation is all that's needed. I contribute to a fairly technical podcast, and I was once asked whether I could release transcripts. It's beyond my means to produce those for every episode, but as a concession I have, ever since, included either existing reference documentation, or I write new documentation on the podcast's website, so that even if a potential listener can't process what I say in the podcast, at least the information I impart isn't lost. It's not the best solution (although admittedly my podcasts aren't always as focused as they could be, so actually reference documentation is probably the better option) but the "answer" to the problem is really easy for me to do, but something I hadn't thought to do until someone asked.

More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources

Sometimes the "right" answer is "no." I've gotten requests for visuals to accompany audio-only content before. While it was possible to do that, it would have required a completely different production and hosting infrastructure, and so the answer truly was "no." However, I was able to respond to the request with a list of resources that were providing similar content along with video. You can't be everything to all people. Knowing your project's, and your own, limitations is important, and it's equally important to respect them.

Open communication

Communication is the starting point for usability and accessibility. When someone reaches out to you because something you're doing isn't accessible to them, that is, strange though it may seem, a marketing success. Somebody wants to engage with your content or your project. That's exciting! Don't pass up those opportunities.

Use open source principles to make your project more accessible for your users.

Image by:

Monsterkoi. Modified by Opensource.com. CC BY-SA 4.0

Accessibility What to read next 8 accessible Linux distributions to try New open source tool catalogs African language resources This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Use Tracee to solve for missing BTF information Alessio Greggi Thu, 09/01/2022 - 03:00 Register or Login to like Register or Login to like

Tracee is a project by Aqua Security for tracing processes at runtime. By tracing processes using Linux eBPF (Berkeley packet filter) technology, Tracee can correlate collected information and identify malicious behavioral patterns.

eBPF

BPF is a system to help in network traffic analysis. The later eBPF system extends classic BPF to improve the programmability of the Linux kernel in different areas, such as network filtering, function hooking, and so on. Thanks to its register-based virtual machine, which is embedded in the kernel, eBPF can execute programs written with a restricted C language without needing to recompile the kernel or load a module. Through eBPF, you can run your program in kernel context and hook various events in the kernel path. To do so, eBPF needs to have deep knowledge about data structures that the kernel is using.

eBPF CO-RE

eBPF interfaces with Linux kernel ABI (application binary interface). Access to kernel structures from eBPF VM depends on the specific Linux kernel release.

eBPF CO-RE (compile once, run everywhere) is the ability to write an eBPF program that will successfully compile, pass kernel verification, and work correctly across different kernel releases without the need to recompile it for each particular kernel.

Ingredients

CO-RE needs a precise synergism of these components:

• BTF (BPF type format) information: Allows the capture of crucial pieces of information about kernel and BPF program types and code, enabling all the other parts of BPF CO-RE puzzle.
   
• Compiler (Clang): Records relocation information. For example, if you were going to access the task_struct->pid field, Clang would record that it was exactly a field named pid of type pid_t residing within a struct task_struct. This system ensures that even if a target kernel has a task_struct layout in which the pid field is moved to a different offset within a task_struct structure, you'll still be able to find it just by its name and type information.
   
• BPF loader (libbpf): Ties BTFs from kernel and BPF programs together to adjust compiled BPF code to specific kernels on target hosts.

So how do these ingredients mix together for a successful recipe?

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Development/build

To make the code portable, the following tricks come into play:

• CO-RE helpers/macros
• BTF-defined maps
• #include "vmlinux.h" (the header file containing all the kernel types)

Run

The kernel must be built with the CONFIG_DEBUG_INFO_BTF=y option in order to provide the /sys/kernel/btf/vmlinux interface that exposes BTF-formatted kernel types. This allows libbpf to resolve and match all the types and fields and update necessary offsets and other relocatable data to make sure that the eBPF program is working properly for the specific kernel on the target host.

The problem

The problem arises when an eBPF program is written to be portable but the target kernel doesn't expose the /sys/kernel/btf/vmlinux interface. For more information, refer to this list of distributions that support BTF.

To load an run an eBPF object in different kernels, the libbpf loader uses the BTF information to calculate field offset relocations. Without the BTF interface, the loader doesn't have the necessary information to adjust the previously recorded types that the program tries to access after processing the object for the running kernel.

Is it possible to avoid this problem?

Use cases

This article explores Tracee, an Aqua Security open source project, that provides a possible solution.

Tracee provides different running modes to adapt itself to the environment conditions. It supports two eBPF integration modes:

• CO-RE: A portable mode, which seamlessly runs on all supported environments
• Non CO-RE: A kernel-specific mode, requiring the eBPF object to be built for the target host

Both of them are implemented in the eBPF C code (pkg/ebpf/c/tracee.bpf.c), where the pre-processing conditional directive takes place. This allows you to compile CO-RE the eBPF binary, passing the -DCORE argument at build time with Clang (take a look at the bpf-core Make target).

In this article, we're going to cover a case of the portable mode when the eBPF binary is built CO-RE, but the target kernel has not been built with CONFIG_DEBUG_INFO_BTF=y option.

To better understand this scenario, it helps to understand what's possible when the kernel doesn't expose BTF-formatted types on sysfs.

No BTF support

If you want to run Tracee on a host without BTF support, there are two options:

1. Build and install the eBPF object for your kernel. This depends on Clang and on the availability of a kernel version-specific kernel-headers package.
    
2. Download the BTF files from BTFHUB for your kernel release and provide it to the tracee-ebpf's loader through the TRACEE_BTF_FILE environment variable.

The first option is not a CO-RE solution. It compiles the eBPF binary, including a long list of kernel headers. That means you need kernel development packages installed on the target system. Also, this solution needs Clang installed on your target machine. The Clang compiler can be resource-heavy, so compiling eBPF code can use a significant amount of resources, potentially affecting a carefully balanced production workload. That said, it's a good practice to avoid the presence of a compiler in your production environment. This could lead to attackers successfully building an exploit and performing a privilege escalation.

The second option is a CO-RE solution. The problem here is that you have to provide the BTF files in your system in order to make Tracee work. The entire archive is nearly 1.3 GB. Of course you can provide just the right BTF file for your kernel release, but that can be difficult when dealing with different kernel releases.

In the end, these possible solutions can also introduce problems, and that's where Tracee works its magic.

A portable solution

With a non-trivial building procedure, the Tracee project compiles a binary to be CO-RE even if the target environment doesn't provide BTF information. This is possible with the embed Go package that provides, at runtime, access to files embedded in the program. During the build, the continuous integration (CI) pipeline downloads, extracts, minimizes, and then embeds BTF files along with the eBPF object inside the tracee-ebpf resultant binary.

Tracee can extract the right BTF file and provide it to libbpf, which in turn loads the eBPF program to run across different kernels. But how can Tracee embed all these BTF files downloaded from BTFHub without weighing too much in the end?

It uses a feature recently introduced in bpftool by the Kinvolk team called BTFGen, available using the bpftool gen min_core_btf subcommand. Given an eBPF program, BTFGen generates reduced BTF files, collecting just what the eBPF code needs for its run. This reduction allows Tracee to embed all these files that are now lighter (just a few kilobytes) and support kernels that don't have the /sys/kernel/btf/vmlinux interface exposed.

Tracee build

Here's the execution flow of the Tracee build:

Image by:

(Alessio Greggi and Massimiliano Giovagnoli, CC BY-SA 4.0)

First, you must build the tracee-ebpf binary, the Go program that loads the eBPF object. The Makefile provides the command make bpf-core to build the tracee.bpf.core.o object with BTF records.

Then STATIC=1 BTFHUB=1 make all builds tracee-ebpf, which has btfhub targeted as a dependency. This last target runs the script 3rdparty/btfhub.sh, which is responsible for downloading the BTFHub repositories:

• btfhub
• btfhub-archive

Once downloaded and placed in the 3rdparty directory, the procedure executes the downloaded script 3rdparty/btfhub/tools/btfgen.sh. This script generates reduced BTF files, tailored for the tracee.bpf.core.o eBPF binary.

The script collects *.tar.xz files from 3rdparty/btfhub-archive/ to uncompress them and finally process them with bpftool, using the following command:

for file in $(find ./archive/${dir} -name *.tar.xz); do
    dir=$(dirname $file)
    base=$(basename $file)
    extracted=$(tar xvfJ $dir/$base)
    bpftool gen min_core_btf ${extracted} dist/btfhub/${extracted} tracee.bpf.core.o
done

This code has been simplified to make it easier to understand the scenario.

Now, you have all the ingredients available for the recipe:

• tracee.bpf.core.o eBPF object
• BTF reduced files (for all kernel releases)
• tracee-ebpf Go source code

At this point, go build is invoked to do its job. Inside the embedded-ebpf.go file, you can find the following code:

//go:embed "dist/tracee.bpf.core.o"
//go:embed "dist/btfhub/*"

Here, the Go compiler is instructed to embed the eBPF CO-RE object with all the BTF-reduced files inside itself. Once compiled, these files will be available using the embed.FS file system. To have an idea of the current situation, you can imagine the binary with a file system structured like this:

dist
├── btfhub
│   ├── 4.19.0-17-amd64.btf
│   ├── 4.19.0-17-cloud-amd64.btf
│   ├── 4.19.0-17-rt-amd64.btf
│   ├── 4.19.0-18-amd64.btf
│   ├── 4.19.0-18-cloud-amd64.btf
│   ├── 4.19.0-18-rt-amd64.btf
│   ├── 4.19.0-20-amd64.btf
│   ├── 4.19.0-20-cloud-amd64.btf
│   ├── 4.19.0-20-rt-amd64.btf
│   └── ...
└── tracee.bpf.core.o

The Go binary is ready. Now to try it out!

Tracee run

Here's the execution flow of the Tracee run:

Image by:

(Alessio Greggi and Massimiliano Giovagnoli, CC BY-SA 4.0)

As the flow chart illustrates, one of the very first phases of tracee-ebpf execution is to discover the environment where it is running. The first condition is an abstraction of the cmd/tracee-ebpf/initialize/bpfobject.go file, specifically where the BpfObject() function takes place. The program performs some checks to understand the environment and make decisions based on it:

1. BPF file given and BTF (vmlinux or env) exists: always load BPF as CO-RE
2. BPF file given but no BTF exists: it is a non CO-RE BPF
3. No BPF file given and BTF (vmlinux or env) exists: load embedded BPF as CO-RE
4. No BPF file given and no BTF available: check embedded BTF files
5. No BPF file given and no BTF available and no embedded BTF: non CO-RE BPF

Here's the code extract:

func BpfObject(config *tracee.Config, kConfig *helpers.KernelConfig, OSInfo *helpers.OSInfo) error {
        ...
        bpfFilePath, err := checkEnvPath("TRACEE_BPF_FILE")
        ...
        btfFilePath, err := checkEnvPath("TRACEE_BTF_FILE")
        ...
        // Decision ordering:
        // (1) BPF file given & BTF (vmlinux or env) exists: always load BPF as CO-RE
        ...
        // (2) BPF file given & if no BTF exists: it is a non CO-RE BPF
        ...
        // (3) no BPF file given & BTF (vmlinux or env) exists: load embedded BPF as CO-RE
        ...
        // (4) no BPF file given & no BTF available: check embedded BTF files
        unpackBTFFile = filepath.Join(traceeInstallPath, "/tracee.btf")
        err = unpackBTFHub(unpackBTFFile, OSInfo)
       
        if err == nil {
                if debug {
                        fmt.Printf("BTF: using BTF file from embedded btfhub: %v\n", unpackBTFFile)
                }
                config.BTFObjPath = unpackBTFFile
                bpfFilePath = "embedded-core"
                bpfBytes, err = unpackCOREBinary()
                if err != nil {
                        return fmt.Errorf("could not unpack embedded CO-RE eBPF object: %v", err)
                }
       
                goto out
        }
        // (5) no BPF file given & no BTF available & no embedded BTF: non CO-RE BPF
        ...
out:
        config.KernelConfig = kConfig
        config.BPFObjPath = bpfFilePath
        config.BPFObjBytes = bpfBytes
       
        return nil
}

This analysis focuses on the fourth case, when eBPF program and BTF files are not provided to tracee-ebpf. At that point, tracee-ebpf tries to load the eBPF program extracting all the necessary files from its embed file system. tracee-ebpf is able to provide the files that it needs to run, even in a hostile environment. It is a sort of high-resilience mode used when none of the conditions have been satisfied.

As you see, BpfObject() calls these functions in the fourth case branch:

• unpackBTFHub()
• unpackCOREBinary()

They extract respectively:

• The BTF file for the underlying kernel
• The BPF CO-RE binary

Unpack the BTFHub

Now take a look starting from unpackBTFHub():

func unpackBTFHub(outFilePath string, OSInfo *helpers.OSInfo) error {
        var btfFilePath string

        osId := OSInfo.GetOSReleaseFieldValue(helpers.OS_ID)
        versionId := strings.Replace(OSInfo.GetOSReleaseFieldValue(helpers.OS_VERSION_ID), "\"", "", -1)
        kernelRelease := OSInfo.GetOSReleaseFieldValue(helpers.OS_KERNEL_RELEASE)
        arch := OSInfo.GetOSReleaseFieldValue(helpers.OS_ARCH)

        if err := os.MkdirAll(filepath.Dir(outFilePath), 0755); err != nil {
                return fmt.Errorf("could not create temp dir: %s", err.Error())
        }

        btfFilePath = fmt.Sprintf("dist/btfhub/%s/%s/%s/%s.btf", osId, versionId, arch, kernelRelease)
        btfFile, err := embed.BPFBundleInjected.Open(btfFilePath)
        if err != nil {
                return fmt.Errorf("error opening embedded btfhub file: %s", err.Error())
        }
        defer btfFile.Close()

        outFile, err := os.Create(outFilePath)
        if err != nil {
                return fmt.Errorf("could not create btf file: %s", err.Error())
        }
        defer outFile.Close()

        if _, err := io.Copy(outFile, btfFile); err != nil {
                return fmt.Errorf("error copying embedded btfhub file: %s", err.Error())

        }

        return nil
}

The function has a first phase where it collects information about the running kernel (osId, versionId, kernelRelease, etc). Then, it creates the directory that is going to host the BTF file (/tmp/tracee by default). It retrieves the right BTF file from the embed file system:

btfFile, err := embed.BPFBundleInjected.Open(btfFilePath)

Finally, it creates and fills the file.

Unpack the CORE Binary

The unpackCOREBinary() function does a similar thing:

func unpackCOREBinary() ([]byte, error) {
        b, err := embed.BPFBundleInjected.ReadFile("dist/tracee.bpf.core.o")
        if err != nil {
                return nil, err
        }

        if debug.Enabled() {
                fmt.Println("unpacked CO:RE bpf object file into memory")
        }

        return b, nil
}

Once the main function BpfObject()returns, tracee-ebpf is ready to load the eBPF binary through libbpfgo. This is done in the initBPF() function, inside pkg/ebpf/tracee.go. Here's the configuration of the program execution:

func (t *Tracee) initBPF() error {
        ...
        newModuleArgs := bpf.NewModuleArgs{
                KConfigFilePath: t.config.KernelConfig.GetKernelConfigFilePath(),
                BTFObjPath:      t.config.BTFObjPath,
                BPFObjBuff:      t.config.BPFObjBytes,
                BPFObjName:      t.config.BPFObjPath,
        }

        // Open the eBPF object file (create a new module)

        t.bpfModule, err = bpf.NewModuleFromBufferArgs(newModuleArgs)
        if err != nil {
                return err
        }
        ...
}

In this piece of code we are initializing the eBPF args filling the libbfgo structure NewModuleArgs{}. Through its BTFObjPath argument, we are able to instruct libbpf to use the BTF file, previously extracted by the BpfObject() function.

At this point, tracee-ebpf is ready to run properly!

Image by:

(Alessio Greggi and Massimiliano Giovagnoli, CC BY-SA 4.0)

eBPF module initialization

Next, during the execution of the Tracee.Init() function, the configured arguments will be used to open the eBPF object file:

Tracee.bpfModule = libbpfgo.NewModuleFromBufferArgs(newModuleArgs)

Initialize the probes:

t.probes, err = probes.Init(t.bpfModule, netEnabled)

Load the eBPF object into kernel:

err = t.bpfModule.BPFLoadObject()

Populate eBPF maps with initial data:

err = t.populateBPFMaps()

And finally, attach eBPF programs to selected events' probes:

err = t.attachProbes() Conclusion

Just as eBPF simplified the way to program the kernel, CO-RE is tackling another barrier. But leveraging such features has some requirements. Fortunately, with Tracee, the Aqua Security team found a way to take advantage of portability in case those requirements can't be satisfied.

At the same time, we're sure that this is only the beginning of a continuously evolving subsystem that will find increasing support over and over, even in different operating systems.

By tracing processes using Linux eBPF (Berkeley packet filter) technology, Tracee can correlate collected information and identify malicious behavioral patterns.

Linux Security and privacy What to read next Using eBPF for network observability in the cloud This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 31 points Italy

Massimiliano Giovagnoli having always been fascinated by mathematics and computers, began his career as a web developer, and with a need to dive into how things work his interests and experience moved to infrastructure design and management.
With growing awareness and experience in operations and site reliability, he's now a cloud solution architect at Clastix, where he is responsible for cloud native infrastructures and to support the development of cloud native products.
Having fallen in love with Linux, Kubernetes, and the OSS ecosystem, started contributing to Falco.
Now he's a maintainer of the Falco's open infra.
He loves his wife, his cats, and the mountains. When he doesn't think about software, he loves to work out.

| Follow maxgio92 Open Enthusiast Author Register or Login to post a comment.

Nitrux 2.4 Released with Linux Kernel 5.19, KDE Gear 22.08, and New Maui Apps  9to5Linux

During the month of August on Phoronix there were 260 original news stories and Linux hardware reviews / benchmark articles written by your's truly. Here is a look back at what excited Linux / open-source readers the most this month...

Open-Source Arc Graphics, Apple M2, Mesa GPU Drivers & Linux 6.0 Excited Readers In August  Phoronix

Linux and Open Source Veterans Sign On to Form CIQ Leadership Team  PR Web

Armbian 22.08 Arrives with Linux 5.19, Wayland by Default for KDE Plasma, PiKVM Support  9to5Linux

Chrome 105 is here for Windows, macOS and Linux  Chrome Unboxed

Microsoft has chosen Linux distro Mariner as sole host OS for Xbox storefronts  Neowin

Kubuntu Focus NX is a compact desktop with Intel Tiger Lake and Kubuntu Linux software  Liliputing