Open-source News

It was over a year ago that AMD initially added the "GFX90A" target to their LLVM AMDGPU compiler back-end while now this week added to the GNU Compiler Collection for the GCC 13 release not due out until next year is its GFX90A support for the GNU toolchain...

'V Rising' comes to Linux and Steam Deck as it sells a million copies  NME

David Airlie this morning sent in the Direct Rendering Manager (DRM) subsystem updates for the Linux 5.19 merge window. Most notable with the DRM display/graphics driver updates for this next kernel version is a lot of work on Intel Arc Graphics DG2/Alchemist in getting that support ready plus initial Raptor Lake enablement. as well as AMD preparing for next-generation CDNA Instinct products and RDNA3 Radeon RX 7000 series graphics cards...

It's been five years already since Red Hat started Stratis as a configuration daemon built atop LVM and XFS in aiming to provide advanced storage functionality in user-space akin to what is offered by the advanced Btrfs and ZFS file-systems...

The 64-bit Arm (AArch64) architecture changes have been merged into the in-development Linux 5.19 kernel...

Security researcher Jason Donenfeld known as the founder of the WireGuard project has recently been focused on modernizing the Linux kernel's random number generator (RNG/random) code. With the Linux 5.19 kernel there is yet more work landing...

Best Watch Softwares for Linux of 2022 May  BollyInside

Google Chrome 102: Critical Security Update For All Windows, Mac & Linux Users  Forbes

'V Rising' comes to Linux and Steam Deck as it sells a million copies  NME

Improve network performance with this open source framework Hifza Khalid Wed, 05/25/2022 - 03:00 2 readers like this 2 readers like this

In the age of high-speed internet, most large information systems are structured as distributed systems with components running on different machines. The performance of these systems is generally assessed by their throughput and response time. When performance is poor, debugging these systems is challenging due to the complex interactions between different subcomponents and the possibility of the problem occurring at various places along the communication path.

On the fastest networks, the performance of a distributed system is limited by the host's ability to generate, transmit, process, and receive data, which is in turn dependent on its hardware and configuration. What if it were possible to tune the network performance of a distributed system using a repository of network benchmark runs and suggest a subset of hardware and OS parameters that are the most effective in improving network performance?

To answer this question, our team used Pbench, a benchmarking and performance analysis framework developed by the performance engineering team at Red Hat. This article will walk step by step through our process of determining the most effective methods and implementing them in a predictive performance tuning tool.

What is the proposed approach?

Given a dataset of network benchmark runs, we propose the following steps to solve this problem.

1. Data preparation: Gather the configuration information, workload, and performance results for the network benchmark; clean the data; and store it in a format that is easy to work with
    
2. Finding significant features: Choose an initial set of OS and hardware parameters and use various feature selection methods to identify the significant parameters
    
3. Develop a predictive model: Develop a machine learning model that can predict network performance for a given client and server system and workload
    
4. Recommend configurations: Given the user's desired network performance, suggest a configuration for the client and the server with the closest performance in the database, along with data showing the potential window of variation in results
    
5. Evaluation: Determine the model's effectiveness using cross-validation, and suggest ways to quantify the improvement due to configuration recommendations

We collected the data for this project using Pbench. Pbench takes as input a benchmark type with its workload, performance tools to run, and hosts on which to execute the benchmark, as shown in the figure below. It outputs the benchmark results, tool results, and the system configuration information for all the hosts.

Image by:

(Hifza Khalid, CC BY-SA 4.0)

Out of the different benchmark scripts that Pbench runs, we used data collected using the uperf benchmark. Uperf is a network performance tool that takes the description of the workload as input and generates the load accordingly to measure system performance.

Data preparation

There are two disjoint sets of data generated by Pbench. The configuration data from the systems under test is stored in a file system. The performance results, along with the workload metadata, are indexed into an Elasticsearch instance. The mapping between the configuration data and the performance results is also stored in Elasticsearch. To interact with the data in Elasticsearch, we used Kibana. Using both of these datasets, we combined the workload metadata, configuration data, and performance results for each benchmark run.

Finding significant features

To select an initial set of hardware specifications and operating system configurations, we used performance-tuning configuration guides and feedback from experts at Red Hat. The goal of this step was to start working with a small set of parameters and refine it with further analysis. The set was based on parameters from almost all major system subcomponents, including hardware, memory, disk, network, kernel, and CPU.

Once we selected the preliminary set of features, we used one of the most common dimensionality-reduction techniques to eliminate the redundant parameters: remove parameters with constant values. While this step eliminated some of the parameters, given the complexity of the relationship between system information and performance, we resolved to use advanced feature selection methods.

Correlation-based feature selection

Correlation is a common measure used to find the association between two features. The features have a high correlation if they are linearly dependent. If the two features increase simultaneously, their correlation is +1; if they decrease concurrently, it is -1. If the two features are uncorrelated, their correlation is close to 0.

We used the correlation between the system configuration and the target variable to identify and cut down insignificant features further. To do so, we calculated the correlation between the configuration parameters and the target variable and eliminated all parameters with a value less than |0.1|, which is a commonly used threshold to identify the uncorrelated pairs.

Feature-selection methods

Since correlation does not imply causation, we needed additional feature-selection methods to extract the parameters affecting the target variables. We could choose between wrapper methods like recursive feature elimination and embedded methods like Lasso (Least Absolute Shrinkage and Selection Operator) and tree-based methods.

We chose to work with tree-based embedded methods for their simplicity, flexibility, and low computational cost compared to wrapper methods. These methods have built-in feature selection methods. Among tree-based methods, we had three options: a classification and regression tree (CART), Random Forest, and XGBoost.

We calculated our final set of significant features for the client and server systems by taking a union of the results received from the three tree-based methods, as shown in the following table.

Parameters client/server Description Advertised_auto-negotation  client If the linked advertised auto-negotiation CPU(s) server Number of logical cores on the machine Network speed server Speed of the ethernet device Model name client Processor model rx_dropped server Packets dropped after entering the computer stack Model name server Processor model System type server Virtual or physical system Develop predictive model

For this step, we used the Random Forest (RF) prediction model since it is known to perform better than CART and is also easier to visualize.

Random Forest (RF) builds multiple decision trees and merges them to get a more stable and accurate prediction. It builds the trees the same way CART does, but to ensure that the trees are uncorrelated to protect each other from their individual errors, it uses a technique known as bagging. Bagging uses random samples from the data with replacement to train the individual trees. Another difference between trees in a Random Forest and a CART decision tree is the choice of features considered for each split. CART considers every possible feature for each split. However, each tree in a Random Forest picks only from a random subset of features. This leads to even more variation among the Random Forest trees.

The RF model was constructed separately for both the target variables.

More for sysadmins Enable Sysadmin blog The Automated Enterprise: A guide to managing IT with automation eBook: Ansible automation for Sysadmins Tales from the field: A system administrator's guide to IT automation eBook: A guide to Kubernetes for SREs and sysadmins Latest sysadmin articles Recommend configurations

For this step, given desired throughput and response time values, along with the workload of interest, our tool searches through the database of benchmark runs to return the configuration with the performance results closest to what the user requires. It also returns the standard deviation for various samples of that run, suggesting potential variation in the actual results.

Evaluation

To evaluate our predictive model, we used a repeated K-Fold cross-validation technique. It is a popular choice to get an accurate estimate of the efficiency of the predictive model.

To evaluate the predictive model with a dataset of 9,048 points, we used k equal to 10 and repeated the cross-validation method three times. The accuracy was calculated using the two metrics given below.

• R2 score: The proportion of the variance in the dependent variable that is predictable from the independent variable(s). Its value varies between -1 and 1.
• Root mean squared error (RMSE): It measures the average squared difference between the estimated values and the actual values and returns its square root.

Based on the above two criteria, the results for the predictive model with throughput and latency as target variables are as follows:

• Throughput (trans/sec):
  • R2 score: 0.984
  • RMSE: 0.012
• Latency (usec):
  • R2 score: 0.930
  • RMSE: 0.025

What does the final tool look like?

We implemented our approach in a tool shown in the following figure. The tool is implemented in Python. It takes as input the dataset containing the information about benchmark runs as a CSV file, including client and server configuration, workload, and the desired values for latency and throughput. The tool uses this information to predict the latency and throughput results for the user's client server system. It then searches through the database of benchmark runs to return the configuration that has performance results closest to what the user requires, along with the standard deviation for that run. The standard deviation is part of the dataset and is calculated using repeated samples for one iteration or run.

Image by:

(Hifza Khalid, CC BY-SA 4.0)

What were the challenges with this approach?

While working on this problem, there were several challenges that we addressed. The first major challenge was gathering benchmark data, which required learning Elasticsearch and Kibana, the two industrial tools used by Red Hat to index, store, and interact with Pbench data. Another difficulty was dealing with the inconsistencies in data, missing data, and errors in the indexed data. For example, workload data for the benchmark runs was indexed in Elasticsearch, but one of the crucial workload parameters, runtime, was missing. For that, we had to write extra code to access it from the raw benchmark data stored on Red Hat servers.

Once we overcame the above challenges, we spent a large chunk of our effort trying out almost all the feature selection techniques available and figuring out a representative set of hardware and OS parameters for network performance. It was challenging to understand the inner workings of these techniques, their limitations, and their applications and analyze why most of them did not apply to our case. Because of space limitations and shortage of time, we did not discuss all of these methods in this article.

Use Pbench to predict throughput and latency for specific workloads.

Networking Databases What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Microsoft brings Linux XDP project to Windows  Neowin

Best Puzzle Games for Linux of 2022 May  BollyInside

Welcomes SoftBank Group to its member ranks

TOKYO, May 25, 2022 – The SODA Foundation, which hosts the SODA Open Data Framework (ODF) for data mobility from edge to core to cloud, today announced two new open source projects: Kahu and Como. Kahu streamlines data protection for Kubernetes and its application data, and Como is a virtual data lake project to enable seamless access to data stored in different clouds. The SODA Foundation also welcomes SoftBank Group as an end-user supporter and key collaboration partner on the Como project.

According to the 2021 SODA Data and Storage Trends Report, two of the top challenges in managing data in containers and cloud-native environments are availability (46%) and management tools (38%).  In direct response to the report findings, the SODA Foundation community collaborated to introduce new tooling options through the Kahu project to improve backup and restore practices critical to data availability.  Furthermore, as enterprises become more data-driven and data growth for some enterprises can exceed 10PB per year, object data management offered by the Como Project will play an important role in performance and scalability requirements for cloud-native environments.

“Data collection, management, and consumption is becoming the new competitive battlefield in IT”, said Steven Tan, chairman, SODA Foundation. “We’re excited to announce Kahu and Como as the latest advances in open source data management and storage. Our 28 members are also excited to welcome the engineers and open source community within SoftBank Group to the Foundation.” 

“Data is the fuel of our global digital economy and harnessing its power requires collaboration on a massive scale”, said Kuniyoshi Suzuki, Senior Director, Cloud Engineering , SoftBank Group.  “Softbank is excited to be joining a community of open source software developers focused on enabling improvements toward data storage, recovery, and retention in cloud environments. We look forward to collaborating with the SODA Foundation and its members, while contributing to the future of this important community.”

New Open Source Releases

In addition to the announcement of Kahu and Como projects, the SODA Foundation also announced the:

• Release of SODA Framework Madagascar v1.7.0: Formerly Open Data Framework (ODF), SODA Framework comprises independent projects initiated by the community to solve common data and storage problems faced by end users. It includes:
  • Terra: a universal SDS controller for connecting storage to Kubernetes, OpenStack, and VMware environments.
  • Delfin: a performance monitor for heterogeneous storage infrastructure in a single pane of glass.
  • Strato: a multi-cloud data controller using a common S3-compatible interface to connect to cloud storage.
  • Kahu : new project to streamline data protection for Kubernetes and application data.

• Expansion of its Eco Project Initiative with the introduction of more open source projects: 

DAOS: a software-defined object store designed from the ground up for massively distributed Non Volatile Memory (NVM), providing features such as transactional non-blocking I/O, advanced data protection with self-healing on top of commodity hardware, end-to-end data integrity, fine-grained data control and elastic storage.

YIG: extends Minio backend storage aggregating multiple Ceph clusters to form a massive storage resource pool that can easily scale up to exabyte (EB) levels with minimal performance disruption.

CubeFS: a cloud-native storage platform used as the underlying storage infrastructure for online applications, database or data processing services and machine learning jobs orchestrated by Kubernetes.

Karmada: a Kubernetes management system that enables organizations to run cloud-native applications across multiple Kubernetes clusters and clouds, with no changes to your applications.

SBK: an open source software framework for the performance benchmarking of any storage system.

Conferences and Survey

• SODACODE: this week, developers from around the world will participate in SODACODE 2022 – the Data & Storage Hackathon on May 25 – 26.  The first-of-its-kind coding event organized by SODA Foundation is open to developers from all levels ranging from beginner to advanced. The hackathon will conclude with project demonstrations, presentation sessions, panel discussions and an award ceremony for the hackathon winners.
• Trend Survey: The SODA Foundation will release its second-annual Data and Storage Trends Survey on June 30, 2022.
• SODACON: a technical conference held by SODA Foundation, will be held this year in Yokohama, Japan on December 7, 2022. The conference will bring together industry leaders, developers and end users to present and discuss the most recent innovations, trends, and concerns as well as practical challenges and solutions in the field of Data and Storage Management in the era of cloud-native, IoT, big data, machine learning, and more.

Additional Resources

• Join the SODA Foundation
• Attend SODACODE 2022 – The Data & Storage Hackathon
• Read the 2021 Data and Storage Trends Report

About the SODA Foundation

Previously OpenSDS, the SODA Foundation is part of the Linux Foundation and includes both open source software and standards to support the increasing need for data autonomy. SODA Foundation Premiere members include China Unicom, Fujitsu, Huawei, NTT Communications and Toyota Motor Corporation. Other members include China Construction Bank Fintech, Click2Cloud, GMO Pepabo, IIJ, MayaData, LinBit, Scality, Sony, Wipro and Yahoo Japan.

Media Contact

info@sodafoundation.io

###

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see its trademark usage page: www.linuxfoundation.org/trademark-usage. Linux is a registered trademark of Linus Torvalds.

The post SODA Foundation Prioritizes Backup and Restore for Containers, Introduces Object Data Management Across Cloud Providers appeared first on Linux Foundation.

The post How to Create Sudo User in openSUSE Linux first appeared on Tecmint: Linux Howtos, Tutorials & Guides .

The sudo command allows a user to administer a Linux system with the security privileges of another user, by default, the superuser or root. In this guide, we will walk you through the process

The post How to Create Sudo User in openSUSE Linux first appeared on Tecmint: Linux Howtos, Tutorials & Guides.

In this month’s customer success highlights, you’ll learn how Red Hat solutions and services helped three organizations achieve their goals in three very different types of digital transformation projects.

First, however, we’d like to give a quick shout out to the 2022 Red Hat Innovation Award winners and honorable mentions.

Malicious Python Repository Package drops Cobalt Strike on Windows, macOS & Linux systems - Urgent Comms  Urgent Communications

David Sterba of SUSE has submitted the ~4k lines of code worth of feature changes for the Btrfs file-system driver in the Linux 5.19 kernel...

Btrfs Gets Some Buttery Good Improvements With Linux 5.19  Phoronix

Ubuntu Users Get New Kernel Security Updates, Three Vulnerabilities Patched  9to5Linux

HP and System76 partner to release high-end Linux laptop  ZDNet