3 practical tips for agile transformation
Agile transformation happens at three levels: team agile, product agile, and organization agile. Helping your team convert to agile is the first and fundamental step in agile transformation, and with good reason. Until you get your people on board with agile, the product of all their hard work can't be agile.
Getting to team agile can be difficult, however. Teams new to agile cite these two factors as common obstacles:
- The team shows strong resistance.
- The team performs when the scrum master is involved, but as soon as the scrum master leaves, the team is thrown into confusion and falls back into old habits.
Overcoming that resistance and sustaining agile workflows takes thoughtful, inclusive leadership. This article will show you how.
Your teams are made of peopleToo often, a scrum master expects a team to respond to instructions like robots. If you step away from the standup for a moment and look at how you're presenting agile to your team, you might find some areas for improvement. Are you dictating to your colleagues how meetings must be? Are you following a script or a recipe to achieve what you've been told agile is? Or are you working with the individuals on the team to determine how to work together to change the way things get done?
3 tips for achieving team agileHere are three tips for those crucial first steps toward team agile.
1. Workshop itThere's nothing wrong with excellent agile practices. The key is in how you get team members to adopt them.
Organize a workshop to discuss current issues with the team. The scrum master is responsible for leading the workshop and providing a structured framework for discussion that encourages team members to put forward ideas and allows members on all sides of a debate to draw their own conclusions. If the team encounters a bottleneck or needs additional information, then the scrum master can provide help and explanation. Always keep in mind, however, that the scrum master's opinion is for reference only and shouldn't be forced on the group. An action plan made this way is co-created by everyone. The whole team is invested in it, which leads to the best possible outcome.
More open source career advice Open source cheat sheets Linux starter kit for developers 7 questions sysadmins should ask a potential employer before taking a job Resources for IT artchitects Cheat sheet: IT job interviews 2. Make it incrementalTell every team member that no matter what conclusions are drawn, the action plan is just an attempt. Schedule regular reviews, collect everyone's feedback, and only continue as long as it's proving effective. When something's not working, make adjustments as needed. Doing this creates a constructive feedback loop that allows team members to be heard and doesn't ask them to commit to wholesale change all at once.
Change can be incremental and iterative. It's still change, and with regular check-ins, it's bound to be change for the better.
[ Read also: Agile transformation: 5 ways to measure progress ]
3. Address benefit redistributionFolks don't like to admit it, but sometimes in an organization there are individuals or departments threatened by agile's transparency. People sometimes take advantage of information asymmetry to occupy important positions and to ensure job security. When faced with agile methodologies, they may feel that they're about to be decentralized, devalued, and weakened. They aren't likely to express this sentiment directly, of course. Instead, they may resist, obstruct, or challenge the new agile methodology, both overtly and covertly.
A scrum master needs to help these individuals see the benefit of agile and recognize that a redistribution of benefits is harmful to no one and better for everyone. Make no mistake: This process is full of challenges, and it can bring up many emotional reactions. Change is often hard, and giving up control of something, even if it means less work, can be stressful. Addressing it directly, and with sensitivity, is vital.
Team agile is called team agile for a reason. It's about a team of people, not individuals hoarding domain control.
Implementation is just the beginningAgile transformation is complex, and most companies have varying degrees of historical baggage to work through. The agile methodology can solve some problems immediately, but at first its primary function is often to expose problems.
You must analyze the issues that agile exposes to find hidden systemic flaws, and you must escalate the issues to promote real problem solving. As with the rest of the process, change is incremental, and it doesn't happen overnight. As long as you're willing to put in the work, however, you can lead your teams to a stronger, more agile state.
This article is adapted from the ZenTao blog and is republished with permission.
Try these tips to help your team make the agile transformation.
Image by:Image by Mapbox Uncharted ERG, CC-BY 3.0 US
Agile What to read next 5 agile mistakes I've made and how to solve them This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Looking into the future of collaborative commons
I read Jeremy Rifkin's book The Zero Marginal Cost Society: The Internet of Things, the Collaborative Commons, and the Eclipse of Capitalism, which has a strong connection to open organization principles, particularly community building. Rifkin also writes about the future of green energy generation and energy use in logistics. In the previous articles in this series, I wrote about the communication internet being joined by the advancement of the energy (producing, sharing, consuming) internet. In this final part of the series, I look at Rifkin's thoughts regarding logistics (moving, storing, sharing) internet, and other economic sectors.
There are public transportation systems like roads, commuter trains, light rail, and buses that are supported by taxes. There are also private transportation options such as private ocean shipping companies, private cars, bicycles, and walking. All these modes of transportation will go through changes with an IoT standardized system that communicates with many moving vehicles. This will make movement of both people and goods more efficient (less waste and more full utilization of capacity). Established protocols will allow firms to collaborate with each other to a more detailed degree. Furthermore, inventory storage will become more efficient. Redundancies and inefficiencies will be identified and reduced.
This can be achieved by a distributed, collaborative, laterally scaled internet communication system, with its open system configuration and commons-style management, as a model for radically transforming global logistics.
Standardization requiredPhysical products need to be embedded in standardized modular containers that can be transported across all the logistics networks. The containers need to be equipped with standardized-protocol smart tags and sensors for identification and sorting.
Open supply network and cooperatives requiredCurrently, private firms have limited internal warehouses and distribution centers, limiting their ability to store and move goods efficiently. Suppose they are on an open supply web that's essentially everywhere and managed by sophisticated analytics and algorithms. In that case, they can store items and route goods more efficiently at any given moment in time.
Furthermore, as 3D printing advances, firms can transport the code for products instead of the physical item itself, reducing a great deal of goods movement and storage. What is needed are universal standards and protocols and a business model similar to a cooperative that is managed regionally, continentally and globally throughout the logistics system on roads, rails, waterways, and air travel. This logistics system will be part of smart cities, smart regions, smart continents and a smart planet.
All of the above requires a great deal of social capital over financial capital if it is to scale laterally rather than vertically. That is where developing commons management systems is a critical requirement.
Open Organization resources Download resources Join the community What is an open organization? How open is your organization? From ownership to sharingRifkin states and feels the desire and need for individually owned automobiles is declining and will continue to decline in the years ahead. The biggest reason is its utilization rate, which might be 5%. Cars sit unused most of the day. Also, the younger generation has more of a sharing mindset over a single owner, exclusive owner mindset. These youths like a diversity of one's experiences over a wide network of various communities.
Looking forward, as car sharing expands, the need for vehicles on the road declines, and reduces carbon emissions. Rifkin thinks that once people start sharing cars, and sharing bicycles, that walking and public transportation will grow. In addition, solar-powered bicycles are coming on the market, increasing their potential and ease of use. Many of the IT-based public bike-sharing operations are run by non-profit organizations.
Car sharing will expand even more when driverless vehicles become available in more and more communities. They will be safer, as they don't get distracted, intoxicated or fall asleep at the wheel, which will drive down auto insurance. These vehicles should be readily available in 8 to 10 years.
Rifkin forecasts that having access to many things will force companies to move away from just selling things and move toward a wide range of solutions for the user. "Collaborative consumption" is on the move now. Sharing will begin with automobiles, bicycles, homes, clothes, tools, toys, and skills within a networked community. People are learning that going into debt just to have things doesn't create happiness, but experiences do. His belief is that society will move from out of control consumption to a sharing, interacting economy. All this will lead to a dependence on social capital and trust over an anonymous market to buy and sell in.
Even recycling is getting into the sharing economy. Why throw it away when others might like to use an item that still has life in it? This is particularly true for children's clothes, as they grow out of their clothes so quickly, and computers, which can be revitalized with a lightweight Linux distribution.
The Economy of AbundanceWhen products and services become very close to being free, the capitalistic system of making a profit breaks down. The motivation to provide goods and services has to be redirected or nothing will be provided.
Rifkin wrote, "Free implies free in two senses of the term: free in price and free from scarcity…Exchange value becomes useless because everyone can secure much of what they need without having to pay for it."
We have all heard the expression, "It's better to give than receive." Well, that's doubly true if you have so much of something that you can't use, spend, or consume it over your lifetime. Most people would rather give something to someone that could fully use it than just throw it away or keep it. The only issue remains is where to give it away that would produce the greatest good, and how to efficiently and effortlessly find the person that needs it?
Some people who have more food than they need mistakenly overconsume. This leads to diabetes, cancers, heart disease, and stroke. That could be true for anything, not just eating. Therefore, wise consumption is important and wise donations will result in a far healthier life. The World Happiness Report is a writing on generosity as an indication of wellbeing. With those thoughts in mind, sharing and donations have their own social capital.
Rifkin writes, "As the poor are lifted out of poverty, they begin to experience happiness. Each advance in income, wealth, and security makes them happier. But here's where it becomes surprising. When individuals reach an income level that provides the basic comforts and securities of life, their level of happiness begins to plateau (in the United States, one study puts it at an income of US $200,000 a year). Additional increases in wealth, and accompanying consumption, triggers diminishing marginal returns in overall happiness, until a point is reached, after which happiness actually reverses course and individuals become less happy."
Individuals in that situation notice superficial behavior from others and seem to be valued only by what can be gained in a material sense. This level of happiness can be seen in nations with a very unbalanced income distribution level. The opposite is true in countries with low income gaps.
Many studies have shown that strict materialist values are connected to depression and substance abuse. Conversely, it is our support and connections with other people that bring us the greatest joy.
The old expression "you can't take it with you" may be true for material things, but humans take joy from interacting with other people. Humans are all social creatures, and their interaction and empathy with others not only bring them joy, but health as well. They are more "other" oriented and feel the warmth of being a "giver".
Imagine this scenario. You visit a museum across town, and you notice that your neighbor is also there. You drove to the museum in your car, and your neighbor arrived by public bus. Because you live next to each other, you offer your neighbor a ride home. Would you consider asking him to pay you for the ride? Of course not. This is true for photographs we share. We don't ask to be paid for sharing a photo. The reason is the social value of offering a ride or a photo is far greater than a financial reward. With abundance and excesses coming in electricity and many other things once not fully available, that level of sharing for free will grow, particularly with people around us that we know personally. This is why the sharing economy is growing, and people are not expecting to exchange something to get financial value or added status.
Rifkin writes that more people are "far more interested in the use value of material things than their exchange value or status. A sharing economy of collaborative prosumers is, by its very nature, a more empathic and less materialistic one." He mentions, "two governing assumptions of economics: that the things we want most in life are scarce, and that our wants are unlimited. In reality, the things we want most are not scarce but infinitely abundant — love, acceptance, and recognition of our humanity."
When abundance replaces scarcity, people are more likely to have less drive to overly consume in fear of what tomorrow might bring. All this will lead to a strong sharing economy in which there is less materialism and more sustainability, less expedient measures, and more empathy. People will think and live more globally in communities and less in capitalistic markets. There will be a great move toward education and learning and less on just getting by.
Looking into the future through peer-to-peer reputation, trust and respectAccording to Rifkin, peer-to-peer reviews and recommendations on on-line shopping websites are more trusted than TV celebrity endorsements. He thinks this will be true in building collaborative commons as well. We could call it mass word of mouth advertising.
Mankind has moved from mythological consciousness (tribal empathy, mythology), to theological consciousness (religious empathy), to ideological consciousness (national and political empathy), and finally to psychological consciousness (global empathy within specialties, global values).
An open organization communityIn these articles, I showed that a new supply network and cooperatives will come back in importance, as it was hundreds of years ago. Also, I presented that the sharing and access of things needed will be more important than ownership. Lastly, far more abundant goods will come off the selling market, like photograph film and telephone services.
With these thoughts in mind, looking into the future, through peer-to-peer reputation, trust and respect will become increasingly important. Also, the contributions of all three sectors (government, the social economy of the commons, and the market) will continue in a hybrid form, although by mid-century, the collaborative commons sector is likely to define much of economic life in society.
Along the way, our ability to empathize with a wide range of people within the global community will be the result. This is what Rifkin calls the "collaborative age", when we all become part of the open organization community. Our wellbeing will be impacted and be influenced by ever larger global communities.
The contributions of all three sectors (government, the social economy of the commons, and the market) will continue in a hybrid form. Eventually, the collaborative commons sector is likely to define much of economic life in society.
The Open Organization What to read next How collaborative commons and open organization principles align Near zero marginal cost societies and the impact on why we work This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Document your source code with Doxygen on Linux
When trying to familiarize yourself with someone else's project, you usually appreciate the comments left behind that help you understand the meaning of their code. In the same way, whenever you are programming, whether for yourself or for others, it is good practice to comment your own code. All programming languages offer a special syntax to mark a word, a line, or a whole section as a comment. Those areas are then ignored by the compiler or interpreter when the source code is processed.
Comments don't take the place of documentation, but there is a way to use your comments to produce documentation easily. Meet Doxygen, an open source tool for generating HTML or LaTeX documentation based on comments in the code. Doxygen enables you to provide a comprehensive overview of the structure of your code without additional effort. While Doxygen is mainly used to document C++, you can use it for many other languages, like C, Objective-C, C#, PHP, Java, Python, and more.
To use Doxygen, you simply comment your source code in a syntax that Doxygen can read. Doxygen then walks through your source files and creates HTML or LaTeX documentation based on those special comments. The C++ example project below will illustrate how the source code is commented and how the documentation is generated from it. The example is available on GitHub, and I will also include references to different sections of the Doxygen manual and documentation.
[ Download now: Doxygen cheat sheet ]
Install Doxygen on LinuxOn Fedora, Doxygen is available as a package. Open a terminal and run:
sudo dnf install doxygenOn Debian-based systems, you can install it by running:
sudo apt-get install doxygenUsageOnce installed, all you need is a project with Doxygen-compatible comments and a Doxyfile, a configuration file that controls the behavior of Doxygen.
Note: If you stick to the related example project on GitHub, you can omit the next step.
If there is no Doxyfile yet, you can simply let Doxygen generate a standard template. To do so, navigate to the root of your project and run:
doxygen -gThe -g stands for generate. You should now notice a newly created file called Doxyfile. You can invoke Doxygen by simply running:
doxygenYou should now notice two newly created folders:
- html/
- latex/
By default, Doxygen outputs LaTeX-formatted documentation as well as HTML-based documentation. In this article, I will focus only on HTML-based documentation. You can find out more about LaTeX output in the official Doxygen documentation, in the Getting started section.
Double click on html/index.html to open the actual HTML documentation. With a blank configuration, it probably looks like the screenshot below:
Image by:(Stephan Avenwedde, CC BY-SA 4.0)
Now it's time to modify the Doxyfile and add some special comments to the source code.
More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles DoxyfileThe Doxyfile allows you to define tons of adjustment possibilities, so I will describe only a very small subset. The settings correspond to the Doxyfile of the example project.
Line 35: Project nameHere you can specify the project name, which will be visible in the header line and the browser tab.
# The PROJECT_NAME tag is a single word (or a sequence of words surrounded by# double-quotes, unless you are using Doxywizard) that should identify the
# project for which the documentation is generated. This name is used in the
# title of most generated pages and in a few other places.
# The default value is: My Project.
PROJECT_NAME = "My Project"Line 47: Project brief description
The brief description will also be shown in the header but in a smaller font.
# Using the PROJECT_BRIEF tag one can provide an optional one line description# for a project that appears at the top of each page and should give viewer a
# quick idea about the purpose of the project. Keep the description short.
PROJECT_BRIEF = "An example of using Doxygen in C++"Line 926: Inclusion of subdirectories
Allow Doxygen to walk recursively through subdirectories to find source and documentation files.
# The RECURSIVE tag can be used to specify whether or not subdirectories should# be searched for input files as well.
# The default value is: NO.
RECURSIVE = YESLine 1769: Disable LaTeX output
If you are just interested in the HTML output, you can disable the LaTeX code generation using this switch.
# If the GENERATE_LATEX tag is set to YES, doxygen will generate LaTeX output.# The default value is: YES.
GENERATE_LATEX = NO
After every change, you can run Doxygen again to check whether your changes have had the desired effect. If you just want to check which modification an existing Doxyfile has, invoke Doxygen with the -x switch:
Image by:(Stephan Avenwedde, CC BY-SA 4.0)
As with the command diff, Doxygen displays only the differences between the actual Doxyfile and the template.
Special commentsDoxygen reads the sources and checks each file for special comments. Based on those comments and keywords, it builds up the HTML documentation. The anatomy of the special comments can be well explained using the header file of the class ByteStream, as shown in the GitHub example linked above.
I will look at the constructor and destructor as an example:
/*! @brief Constructor which takes an external buffer to operate on*
* The specified buffer already exist.
* Memory and size can be accessed by buffer() and size().
*
* @param[in] pBuf Pointer to existing buffer
* @param[in] size Size of the existing buffer
*/
ByteStream(char* pBuf, size_t size) noexcept;
There are different flavors of formatting a special comment block. I prefer to start the comment in the Qt-style (/*!) and add an asterisk (*) before each line. The block then ends with an asterisk followed by a forward slash (*/). To get an overview of the different style options, refer to the Doxygen manual, in the section Documenting the code.
Comments in Doxygen are divided into two sections, a brief description and a detailed description. Both sections are optional. In the code sample above, the comment block refers to the following line of code, the declaration of a constructor. The sentence behind the @brief will be shown in the compact class overview:
Image by:(Stephan Avenwedde, CC BY-SA 4.0)
After a blank line (blank lines are treated as paragraph separators), the actual documentation for the constructor begins. With the @param[in/out] keyword, you can mark the arguments passed to the constructor, and Doxygen will make a clear argument list from it:
Image by:(Stephan Avenwedde, CC BY-SA 4.0)
Note that Doxygen automatically creates a link to the mentioned buffer() and size() method in the comment. In contrast, the comment before the destructor declaration won't have any effect on Doxygen as it is not recognized as a special comment:
// Destructor~ByteStream();
Now you have seen 90% of the magic. By using a slightly modified syntax for your comments, you can convert them into special comments that Doxygen can read. Furthermore, by using a few keywords, you can advance the formatting. In addition, Doxygen has some special features, which I will highlight in the following section.
FeaturesMost of the work is already done via your regular commenting on the source code. But with a few tweaks, you can easily enhance the output of Doxygen.
MarkdownFor advanced formatting, Doxygen supports Markdown syntax and HTML commands. There is a Markdown cheat sheet available in the download section of opensource.com.
MainpageAside from your customized header, you will get a mostly empty page when you open html/index.html. You can add some meaningful content to this empty space by using specific keywords. Because the main page is usually not dedicated to a particular source code file, you can add an ordinary text file containing the content for the main page into the root of your project. You can see this in the example on GitHub. The comments in there produce the following output:
Image by:(Stephan Avenwedde, CC BY-SA 4.0)
Automatic link generationAs noted above, Doxygen automatically figures out when you are referring to an existing part of the code and creates a link to the related documentation. Be aware that automatic link creation only works if the part you refer to is documented as well.
More information can be found in the official documentation, under Automatic link generation.
GroupsThe ByteStream class has overloaded stream operators for writing (<<) and reading (>>). In the class overview in the Special comments section above, you can see that the operator declarations are grouped as Writing and Reading. These groups are defined and named in the ByteStream header file.
Grouping is done using a special syntax: You start a group with @{ and end it with }@. All members inside those marks belong to this group. In the header ByteStream.h it is implemented as follows:
/** @name Writing* Operators for writing to the stream
* @{
*/
(...)
/** @}
* @name Reading
* Operators for reading from the stream
* @{
*/
(...)
/** @} */
You can find more information about grouping in the Doxygen documentation, under Grouping.
LLVM SupportIf you are building with Clang, you can apply the flag -Wdocumentation to the build process to let Clang check your special comments. You can find more information about this feature in the LLVM Users Manual or in Dmitri Gribenko's presentation, both on the Clang website.
Where Doxygen is usedDoxygen was first released in 1997, so it has been already around for some years. Despite its age, many projects use Doxygen to create their documentation. Some examples are NASA's F Prime flight software framework, the image processing library OpenCV, and the package manager RPM. You can also find the Doxygen syntax in other areas, like in the documentation standards of the content management platform Drupal.
A caveat: One drawback of using Doxygen is that it outputs HTML documentation with the look and feel of web pages from the nineties. It is also hard to depict the architecture of meta and template programming using Doxygen. For those cases, you would probably choose Sphinx over Doxygen. Download the Doxygen cheat sheet now.
This widely used open source tool can generate documentation from your comments.
Image by:Internet Archive Book Images. Modified by Opensource.com. CC BY-SA 4.0
Documentation Cheat sheets Linux Text editors What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Write C applications using Vely on Linux
Vely is a tool for writing web and command-line applications in C. Vely combines high performance and the low footprint associated with C programming with ease of use and improved safety reminiscent of languages like PHP. It's free and open source software, and licensed under GPLv3 and LGPL 3 for libraries, so you can even build commercial software with it.
Vely works on major Linux distributions and processor architectures. You can use webservers, such as Apache, Nginx, or others, and databases such as MariaDB, PostgreSQL, and SQLite.
You can use Vely for web applications, command-line programs, as middleware, database applications, services software, data integration, IoT (Internet of Things), and anywhere else. It's well suited for the cloud, works easily in a container, and, due to low resource requirements, it's also a good choice when memory and processing power are at a premium.
Install VelyTo try Vely, install the Apache webserver and MariaDB database. You can use a different webserver and database, and the setup would be similar, but in this example, I use Apache and MariaDB.
Next, install Vely. On Linux use a package manager such as dnf or apt.
Stock tickers projectThis example saves the names of stock tickers and their prices so you can view them in a list.
Start by creating stock.v file, and paste this code into it:
#include "vely.h"void stock() {
out-header default
@
@
input-param action
input-param stock_name
input-param stock_price
if (!strcmp (action, "add")) {
// Add to stock table, update if stock exists
run-query#add_data@db = "insert into stock (stock_name,\
stock_price) values ('%s', '%s') on duplicate key \
update stock_price='%s'" : stock_name, stock_price, \
stock_price
end-query
error#add_data to define err
if (strcmp (err, "0")) {
report-error "Cannot update stock price, error [%s]", err
}
@
@Stock price updated!
@
} else if (!strcmp (action, "show")) {
// Show stock names and values
@
@
@Stock name
@Stock price
@
run-query#show_data@db = "select stock_name, \
stock_price from stock" output stock_name, \
stock_price
@
@
query-result#show_data, stock_name
@
@
query-result#show_data, stock_price
@
@
end-query
@
} else {
@Unrecognized request!
}
@
@
}Build the database
For this example, create a database named dbstock, owned by user vely with the password your_password. These are arbitrary names, and in real life, you can use whatever values you want, as long as they're consistent throughout your code.
First, log in to the MariaDB database as root and execute this:
CREATE DATABASE IF NOT EXISTS dbstock;FLUSH privileges;
CREATE USER IF NOT EXISTS vely@localhost IDENTIFIED BY 'your_password';
FLUSH privileges;
GRANT ALL privileges ON dbstock.* TO vely@localhost;
FLUSH privileges;
exit;
Now log in to MariaDB again and set the current database:
$ mysql -u vely -pyour_passwordNow you can create the database objects needed for the application. You need a stock table in the dbstock database for this example.
USE dbstock;CREATE TABLE IF NOT EXISTS stock (stock_name VARCHAR(100) PRIMARY KEY, stock_price BIGINT);
Finally, create a database configuration file named db so that your application can log into the database. You must call it db because that's what the code in stock.v uses. For instance:
[...]run-query#add_data@db = "insert into stock ..."
[...]
The database name is preceded by the @ sign, in this case, @db, so the name of the database configuration file is db. As with other values, you can name your database configuration file whatever you want, as long as your code is consistent.
Here's the configuration for the db file:
[client]user=vely
password=your_password
database=dbstock
The above is a standard MariaDB client options file. Vely uses native database connectivity, so you can specify any options a given database allows.
Programming and development Red Hat Developers Blog Programming cheat sheets Try for free: Red Hat Learning Subscription eBook: An introduction to programming with Bash Bash shell scripting cheat sheet eBook: Modernizing Enterprise Java Build the applicationNext, you can create your Vely application. For this example, you're going to create a web app called stockapp:
$ sudo vf -i -u $(whoami) stockappThis creates an application home under the Vely directory (/var/lib/vv) and performs the required application setup steps for you.
To build your application, use the vv command:
$ vv -q --db=mariadb:db stockappHere's what each option means:
- -q builds an application
- --db specifies the database to be used (mariadb:db, as specified in your configuration file)
- stockapp is the application name
You can actually use any number of databases and different vendors in your application. This example is simple, though, so you only need one database. Vely has many other useful options you can use, but this is sufficient for now.
Configure web accessTo access your application via a web browser or various web clients, you need to set up a webserver. It can be Apache, Nginx, or any other server that supports FastCGI proxying (most, if not all, webservers and load balancers do this). Here, I will set up Apache, but the setup is similar for other webservers.
The proxy and proxy_fcgi modules are installed and enabled by default on the Fedora install of the Apache web server, but you must enable them on Debian-based systems (like Ubuntu):
$ sudo a2enmod proxy$ sudo a2enmod proxy_fcgi
$ sudo systemctl restart apache2
If you're not on a Debian-based system, you can enable an Apache module by adding it to the Apache configuration file or in a file in the /etc/httpd/conf.modules.d/ directory, depending on your distribution's configuration.
Next, open your Apache configuration file in a text editor. For example, on a Debian-based system:
$ sudo vi /etc/apache2/apache2.confOn a Fedora system (including Red Hat Enterprise Linux and CentOS):
$ sudo vi /etc/httpd/conf/httpd.confAdd this line to the end of the file:
ProxyPass "/stockapp" unix:///var/lib/vv/stockapp/sock/sock|fcgi://localhost/stockappDepending on your webserver configuration, there may be a better place to add the ProxyPass directive. For this example, though, the above is sufficient.
Save the file and restart the webserver. On Fedora-based systems:
$ sudo systemctl restart httpdOn Debian-based systems:
$ sudo systemctl restart apache2In this case, you're connecting to your application through a socket, but you can use a TCP port instead (which comes in handy when your application resides in a container or something similar).
Run the applicationStart the application server for your application:
$ vf stockappBy default, this runs anywhere from 0 to 20 server processes for your application, depending on the load. When the user load is low, your application uses virtually no memory at all.
That was it! Navigate to http://127.0.0.1/stockapp?req=stock&action=add&stock_name=XYZ&stock_pri… in your web browser to see the application.
You've just updated the stock price for ticker "XYZ" to 440. Try different tickers and prices to build a list of stocks, which you can view with the URL http://127.0.0.1/stockapp?req=stock&action=show.
Congratulations, you've created your first Vely application, reverse proxied behind a web server.
You can also view the output without a graphical browser by using curl:
$ curl -s \"http://127.0.0.1/stockapp?req=stock&action=add&stock_name=XYZ&stock_price=440"
$ curl -s "http://127.0.0.1/stockapp?req=stock&action=show"Run the application from the terminal
You can run your application from the terminal, too. A terminal command is always made along with the FastCGI application server, and it's named the same as your application (in this case, stockapp). It works exactly the same as the web app. You can write some requests to your application to be fulfilled as web requests and others to run from the command-line. To do that, you provide the request as environment variables. For instance, to output the list of stocks as HTML, type:
$ export REQUEST_METHOD=GET$ export QUERY_STRING="req=stock&action=show"
$ /var/lib/vv/bld/stockapp/stockapp
To suppress HTTP headers, use:
$ export VV_SILENT_HEADER=yes$ /var/lib/vv/bld/stockapp/stockappHow Vely works
Your application works by processing requests and sending back replies. A request is one of two HTTP methods: GET or POST.
A request always has a parameter req. In the example here, its value is stock. That means source code compiled from file stock.v is called automatically to handle such a request.
A source file like this can do many different things, all grouped logically under a single request. Here, you have another parameter action, which can have a value of add (to add or update a stock) or show (to show a list of stocks). You specify stock_name and stock_price parameters when adding or updating. Pretty easy stuff. Other than req, you can choose parameter names however you wish.
Looking at the code in stock.v, it's simple to follow. You use the input-param construct to get the values for your input parameters. Yes, those strange things in the C code that aren't C are Vely language constructs, and they do lots of useful stuff for you, such as run-query, which (as you might expect from the name) runs your queries. An easy one is @, which is an output construct. String handling is made simple and reliable without worrying about buffer overruns. Check out the full reference of Vely constructs to understand Vely's capabilities.
Vely converts all constructs in your code into pure C and makes a native executable that is very small and fast. Your application runs as several FastCGI server processes, which stay resident in memory while accepting and processing requests. All of these processes work in parallel.
For more info, see how Vely works and read more about Vely architecture.
Manage strings and memoryVely has automatic garbage collection for all of its constructs. In fact, most of the time, you shouldn't need to free memory at all, so application development is even simpler. Leave that to Vely and enjoy computing free of memory leaks and far fewer memory issues than you might expect. String constructs such as write-string make it safe, fast, and easy to create complex strings just as they do simple ones.
FastCGI program managerEven if you don't want to develop your own applications with Vely, you can use vf, Vely's FastCGI program manager, with any generic FastCGI program, not just those created with Vely.
Want to learn more about Vely?I sometimes get asked about the project name. Vely is short for Vel(ocit)y. It's fast to program with, fast to understand code and maintain, and fast (and small!) at run time. It's even easy to containerize.
Check out the documentation at vely.dev, which features downloads and examples that go beyond the introduction this article provides.
Vely is an open source tool for writing web and command-line applications in C on major Linux distributions.
Image by:Ray Smith
Programming What to read next How the C programming language has grown Install MariaDB or MySQL on Linux This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Migrate databases to Kubernetes using Konveyor
Kubernetes Database Operator is useful for building scalable database servers as a database (DB) cluster. But because you have to create new artifacts expressed as YAML files, migrating existing databases to Kubernetes requires a lot of manual effort. This article introduces a new open source tool named Konveyor Tackle-DiVA-DOA (Data-intensive Validity Analyzer-Database Operator Adaptation). It automatically generates deployment-ready artifacts for database operator migration. And it does that through datacentric code analysis.
What is Tackle-DiVA-DOA?Tackle-DiVA-DOA (DOA, for short) is an open source datacentric database configuration analytics tool in Konveyor Tackle. It imports target database configuration files (such as SQL and XML) and generates a set of Kubernetes artifacts for database migration to operators such as Zalando Postgres Operator.
Image by:(Yasuharu Katsuno and Shin Saito, CC BY-SA 4.0)
DOA finds and analyzes the settings of an existing system that uses a database management system (DBMS). Then it generates manifests (YAML files) of Kubernetes and the Postgres operator for deploying an equivalent DB cluster.
Image by:(Yasuharu Katsuno and Shin Saito, CC BY-SA 4.0)
More on Kubernetes What is Kubernetes? Free online course: Containers, Kubernetes and Red Hat OpenShift technical over… eBook: Storage Patterns for Kubernetes Test drive OpenShift hands-on An introduction to enterprise Kubernetes How to explain Kubernetes in plain terms eBook: Running Kubernetes on your Raspberry Pi homelab Kubernetes cheat sheet eBook: A guide to Kubernetes for SREs and sysadmins Latest Kubernetes articlesDatabase settings of an application consist of DBMS configurations, SQL files, DB initialization scripts, and program codes to access the DB.
- DBMS configurations include parameters of DBMS, cluster configuration, and credentials. DOA stores the configuration to postgres.yaml and secrets to secret-db.yaml if you need custom credentials.
- SQL files are used to define and initialize tables, views, and other entities in the database. These are stored in the Kubernetes ConfigMap definition cm-sqls.yaml.
- Database initialization scripts typically create databases and schema and grant users access to the DB entities so that SQL files work correctly. DOA tries to find initialization requirements from scripts and documents or guesses if it can't. The result will also be stored in a ConfigMap named cm-init-db.yaml.
- Code to access the database, such as host and database name, is in some cases embedded in program code. These are rewritten to work with the migrated DB cluster.
DOA is expected to run within a container and comes with a script to build its image. Make sure Docker and Bash are installed on your environment, and then run the build script as follows:
$ cd /tmp$ git clone https://github.com/konveyor/tackle-diva.git
$ cd tackle-diva/doa
$ bash util/build.sh
…
docker image ls diva-doa
REPOSITORY TAG IMAGE ID CREATED SIZE
diva-doa 2.2.0 5f9dd8f9f0eb 14 hours ago 1.27GB
diva-doa latest 5f9dd8f9f0eb 14 hours ago 1.27GB
This builds DOA and packs as container images. Now DOA is ready to use.
The next step executes a bundled run-doa.sh wrapper script, which runs the DOA container. Specify the Git repository of the target database application. This example uses a Postgres database in the TradeApp application. You can use the -o option for the location of output files and an -i option for the name of the database initialization script:
$ cd /tmp/tackle-diva/doa$ bash run-doa.sh -o /tmp/out -i start_up.sh \
https://github.com/saud-aslam/trading-app
[OK] successfully completed.
The /tmp/out/ directory and /tmp/out/trading-app, a directory with the target application name, are created. In this example, the application name is trading-app, which is the GitHub repository name. Generated artifacts (the YAML files) are also generated under the application-name directory:
$ ls -FR /tmp/out/trading-app//tmp/out/trading-app/:
cm-init-db.yaml cm-sqls.yaml create.sh* delete.sh* job-init.yaml postgres.yaml test/
/tmp/out/trading-app/test:
pod-test.yaml
The prefix of each YAML file denotes the kind of resource that the file defines. For instance, each cm-*.yaml file defines a ConfigMap, and job-init.yaml defines a Job resource. At this point, secret-db.yaml is not created, and DOA uses credentials that the Postgres operator automatically generates.
Now you have the resource definitions required to deploy a PostgreSQL cluster on a Kubernetes instance. You can deploy them using the utility script create.sh. Alternatively, you can use the kubectl create command:
$ cd /tmp/out/trading-app$ bash create.sh # or simply “kubectl apply -f .”
configmap/trading-app-cm-init-db created
configmap/trading-app-cm-sqls created
job.batch/trading-app-init created
postgresql.acid.zalan.do/diva-trading-app-db created
The Kubernetes resources are created, including postgresql (a resource of the database cluster created by the Postgres operator), service, rs, pod, job, cm, secret, pv, and pvc. For example, you can see four database pods named trading-app-*, because the number of database instances is defined as four in postgres.yaml.
$ kubectl get all,postgresql,cm,secret,pv,pvcNAME READY STATUS RESTARTS AGE
…
pod/trading-app-db-0 1/1 Running 0 7m11s
pod/trading-app-db-1 1/1 Running 0 5m
pod/trading-app-db-2 1/1 Running 0 4m14s
pod/trading-app-db-3 1/1 Running 0 4m
NAME TEAM VERSION PODS VOLUME CPU-REQUEST MEMORY-REQUEST AGE STATUS
postgresql.acid.zalan.do/trading-app-db trading-app 13 4 1Gi 15m Running
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/trading-app-db ClusterIP 10.97.59.252 <none> 5432/TCP 15m
service/trading-app-db-repl ClusterIP 10.108.49.133 <none> 5432/TCP 15m
NAME COMPLETIONS DURATION AGE
job.batch/trading-app-init 1/1 2m39s 15m
Note that the Postgres operator comes with a user interface (UI). You can find the created cluster on the UI. You need to export the endpoint URL to open the UI on a browser. If you use minikube, do as follows:
$ minikube service postgres-operator-uiThen a browser window automatically opens that shows the UI.
Image by:(Yasuharu Katsuno and Shin Saito, CC BY-SA 4.0)
Now you can get access to the database instances using a test pod. DOA also generated a pod definition for testing.
$ kubectl apply -f /tmp/out/trading-app/test/pod-test.yaml # creates a test Podpod/trading-app-test created
$ kubectl exec trading-app-test -it -- bash # login to the pod
The database hostname and the credential to access the DB are injected into the pod, so you can access the database using them. Execute the psql metacommand to show all tables and views (in a database):
# printenv DB_HOST; printenv PGPASSWORD(values of the variable are shown)
# psql -h ${DB_HOST} -U postgres -d jrvstrading -c '\dt'
List of relations
Schema | Name | Type | Owner
--------+----------------+-------+----------
public | account | table | postgres
public | quote | table | postgres
public | security_order | table | postgres
public | trader | table | postgres
(4 rows)
# psql -h ${DB_HOST} -U postgres -d jrvstrading -c '\dv'
List of relations
Schema | Name | Type | Owner
--------+-----------------------+------+----------
public | pg_stat_kcache | view | postgres
public | pg_stat_kcache_detail | view | postgres
public | pg_stat_statements | view | postgres
public | position | view | postgres
(4 rows)
After the test is done, log out from the pod and remove the test pod:
# exit$ kubectl delete -f /tmp/out/trading-app/test/pod-test.yaml
Finally, delete the created cluster using a script:
$ bash delete.shWelcome to Konveyor Tackle world!To learn more about application refactoring, you can check out the Konveyor Tackle site, join the community, and access the source code on GitHub.
Konveyor Tackle-DiVA-DOA helps database engineers easily migrate database servers to Kubernetes.
Kubernetes Upstream Communities What to read next Refactor your applications to Kubernetes This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 31 points | Follow shinsa82 Open Enthusiast Author Register or Login to post a comment.Improve network performance with this open source framework
In the age of high-speed internet, most large information systems are structured as distributed systems with components running on different machines. The performance of these systems is generally assessed by their throughput and response time. When performance is poor, debugging these systems is challenging due to the complex interactions between different subcomponents and the possibility of the problem occurring at various places along the communication path.
On the fastest networks, the performance of a distributed system is limited by the host's ability to generate, transmit, process, and receive data, which is in turn dependent on its hardware and configuration. What if it were possible to tune the network performance of a distributed system using a repository of network benchmark runs and suggest a subset of hardware and OS parameters that are the most effective in improving network performance?
To answer this question, our team used Pbench, a benchmarking and performance analysis framework developed by the performance engineering team at Red Hat. This article will walk step by step through our process of determining the most effective methods and implementing them in a predictive performance tuning tool.
What is the proposed approach?Given a dataset of network benchmark runs, we propose the following steps to solve this problem.
- Data preparation: Gather the configuration information, workload, and performance results for the network benchmark; clean the data; and store it in a format that is easy to work with
- Finding significant features: Choose an initial set of OS and hardware parameters and use various feature selection methods to identify the significant parameters
- Develop a predictive model: Develop a machine learning model that can predict network performance for a given client and server system and workload
- Recommend configurations: Given the user's desired network performance, suggest a configuration for the client and the server with the closest performance in the database, along with data showing the potential window of variation in results
- Evaluation: Determine the model's effectiveness using cross-validation, and suggest ways to quantify the improvement due to configuration recommendations
We collected the data for this project using Pbench. Pbench takes as input a benchmark type with its workload, performance tools to run, and hosts on which to execute the benchmark, as shown in the figure below. It outputs the benchmark results, tool results, and the system configuration information for all the hosts.
Image by:(Hifza Khalid, CC BY-SA 4.0)
Out of the different benchmark scripts that Pbench runs, we used data collected using the uperf benchmark. Uperf is a network performance tool that takes the description of the workload as input and generates the load accordingly to measure system performance.
Data preparationThere are two disjoint sets of data generated by Pbench. The configuration data from the systems under test is stored in a file system. The performance results, along with the workload metadata, are indexed into an Elasticsearch instance. The mapping between the configuration data and the performance results is also stored in Elasticsearch. To interact with the data in Elasticsearch, we used Kibana. Using both of these datasets, we combined the workload metadata, configuration data, and performance results for each benchmark run.
Finding significant featuresTo select an initial set of hardware specifications and operating system configurations, we used performance-tuning configuration guides and feedback from experts at Red Hat. The goal of this step was to start working with a small set of parameters and refine it with further analysis. The set was based on parameters from almost all major system subcomponents, including hardware, memory, disk, network, kernel, and CPU.
Once we selected the preliminary set of features, we used one of the most common dimensionality-reduction techniques to eliminate the redundant parameters: remove parameters with constant values. While this step eliminated some of the parameters, given the complexity of the relationship between system information and performance, we resolved to use advanced feature selection methods.
Correlation-based feature selectionCorrelation is a common measure used to find the association between two features. The features have a high correlation if they are linearly dependent. If the two features increase simultaneously, their correlation is +1; if they decrease concurrently, it is -1. If the two features are uncorrelated, their correlation is close to 0.
We used the correlation between the system configuration and the target variable to identify and cut down insignificant features further. To do so, we calculated the correlation between the configuration parameters and the target variable and eliminated all parameters with a value less than |0.1|, which is a commonly used threshold to identify the uncorrelated pairs.
Feature-selection methodsSince correlation does not imply causation, we needed additional feature-selection methods to extract the parameters affecting the target variables. We could choose between wrapper methods like recursive feature elimination and embedded methods like Lasso (Least Absolute Shrinkage and Selection Operator) and tree-based methods.
We chose to work with tree-based embedded methods for their simplicity, flexibility, and low computational cost compared to wrapper methods. These methods have built-in feature selection methods. Among tree-based methods, we had three options: a classification and regression tree (CART), Random Forest, and XGBoost.
We calculated our final set of significant features for the client and server systems by taking a union of the results received from the three tree-based methods, as shown in the following table.
Parameters client/server Description Advertised_auto-negotation client If the linked advertised auto-negotiation CPU(s) server Number of logical cores on the machine Network speed server Speed of the ethernet device Model name client Processor model rx_dropped server Packets dropped after entering the computer stack Model name server Processor model System type server Virtual or physical system Develop predictive modelFor this step, we used the Random Forest (RF) prediction model since it is known to perform better than CART and is also easier to visualize.
Random Forest (RF) builds multiple decision trees and merges them to get a more stable and accurate prediction. It builds the trees the same way CART does, but to ensure that the trees are uncorrelated to protect each other from their individual errors, it uses a technique known as bagging. Bagging uses random samples from the data with replacement to train the individual trees. Another difference between trees in a Random Forest and a CART decision tree is the choice of features considered for each split. CART considers every possible feature for each split. However, each tree in a Random Forest picks only from a random subset of features. This leads to even more variation among the Random Forest trees.
The RF model was constructed separately for both the target variables.
More for sysadmins Enable Sysadmin blog The Automated Enterprise: A guide to managing IT with automation eBook: Ansible automation for Sysadmins Tales from the field: A system administrator's guide to IT automation eBook: A guide to Kubernetes for SREs and sysadmins Latest sysadmin articles Recommend configurationsFor this step, given desired throughput and response time values, along with the workload of interest, our tool searches through the database of benchmark runs to return the configuration with the performance results closest to what the user requires. It also returns the standard deviation for various samples of that run, suggesting potential variation in the actual results.
EvaluationTo evaluate our predictive model, we used a repeated K-Fold cross-validation technique. It is a popular choice to get an accurate estimate of the efficiency of the predictive model.
To evaluate the predictive model with a dataset of 9,048 points, we used k equal to 10 and repeated the cross-validation method three times. The accuracy was calculated using the two metrics given below.
- R2 score: The proportion of the variance in the dependent variable that is predictable from the independent variable(s). Its value varies between -1 and 1.
- Root mean squared error (RMSE): It measures the average squared difference between the estimated values and the actual values and returns its square root.
Based on the above two criteria, the results for the predictive model with throughput and latency as target variables are as follows:
- Throughput (trans/sec):
- R2 score: 0.984
- RMSE: 0.012
- Latency (usec):
- R2 score: 0.930
- RMSE: 0.025
We implemented our approach in a tool shown in the following figure. The tool is implemented in Python. It takes as input the dataset containing the information about benchmark runs as a CSV file, including client and server configuration, workload, and the desired values for latency and throughput. The tool uses this information to predict the latency and throughput results for the user's client server system. It then searches through the database of benchmark runs to return the configuration that has performance results closest to what the user requires, along with the standard deviation for that run. The standard deviation is part of the dataset and is calculated using repeated samples for one iteration or run.
Image by:(Hifza Khalid, CC BY-SA 4.0)
What were the challenges with this approach?While working on this problem, there were several challenges that we addressed. The first major challenge was gathering benchmark data, which required learning Elasticsearch and Kibana, the two industrial tools used by Red Hat to index, store, and interact with Pbench data. Another difficulty was dealing with the inconsistencies in data, missing data, and errors in the indexed data. For example, workload data for the benchmark runs was indexed in Elasticsearch, but one of the crucial workload parameters, runtime, was missing. For that, we had to write extra code to access it from the raw benchmark data stored on Red Hat servers.
Once we overcame the above challenges, we spent a large chunk of our effort trying out almost all the feature selection techniques available and figuring out a representative set of hardware and OS parameters for network performance. It was challenging to understand the inner workings of these techniques, their limitations, and their applications and analyze why most of them did not apply to our case. Because of space limitations and shortage of time, we did not discuss all of these methods in this article.
Use Pbench to predict throughput and latency for specific workloads.
Networking Databases What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.12 essential Linux commands for beginners
When operating on the Linux command line, it is easy to get disoriented, which can have disastrous consequences. I once issued a remove command before realizing that I'd moved the boot directory of my computer. I learned to use the pwd command to know exactly which part of the file system I was in (and these days, there are command projects, like trashy and trash-cli, that serve as intermediates when removing files).
When I was new to Linux, I had a cheat sheet that hung over my desk to help me remember those commands as I managed my Linux servers. It was called the 101 commands for Linux cheat sheet. As I became more familiar with these commands, I became more proficient with server administration.
Here are 12 Linux commands I find most useful.
1. Print working directory (pwd)The pwd command prints your working directory. In other words, it outputs the path of the directory you are currently working in. There are two options: --logical to display your location with any symlinks and --physical to display your location after resolving any symlinks.
2. Make directory (mkdir)Making directories is easy with the mkdir command. The following command creates a directory called example unless example already exists:
$ mkdir exampleYou can make directories within directories:
$ mkdir -p example/one/twoIf directories example and one already exist, only directory two is created. If none of them exist, then three nested directories are created.
3. List (ls)Coming from MS-DOS, I was used to listing files with the dir command. I don't recall working on Linux at the time, although today, dir is in the GNU Core Utilities package. Most people use the ls command to display the files, along with all their properties, are in a directory. The ls command has many options, including -l to view a long listing of files, displaying the file owner and permissions.
4. Change directory (cd)It is often necessary to change directories. That's the cd command's function. For instance, this example takes you from your home directory into the Documents directory:
$ cd DocumentsYou can quickly change to your home directory with cd ~ or just cd on most systems. You can use cd .. to move up a level.
More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles 5. Remove a file (rm)Removing files is inherently dangerous. Traditionally, the Linux terminal has no Trash or Bin like the desktop does, so many terminal users have the bad habit of permanently removing data they believe they no longer need. There's no "un-remove" command, though, so this habit can be problematic should you accidentally delete a directory containing important data.
A Linux system provides rm and shred for data removal. To delete file example.txt, type the following:
$ rm example.txtHowever, it's much safer to install a trash command, such as trashy or trash-cli. Then you can send files to a staging area before deleting them forever:
$ trash example.txt6. Copy a file (cp)Copy files with the cp command. The syntax is copy from-here to-there. Here's an example:
$ cp file1.txt newfile1.txtYou can copy entire directories, too:
$ cp -r dir1 newdirectory7. Move and rename a file (mv)Renaming and moving a file is functionally the same process. When you move a file, you take a file from one directory and put it into a new one. When renaming a file, you take a file from one directory and put it back into the same directory or a different directory, but with a new name. Either way, you use the mv command:
$ mv file1.txt file_001.txt8. Create an empty file (touch)Easily create an empty file with the touch command:
$ touch one.txt$ touch two.txt
$ touch three.md9. Change permissions (chmod)
Change the permissions of a file with the chmod command. One of the most common uses of chmod is making a file executable:
$ chmod +x myfileThis example is how you give a file permission to be executed as a command. This is particularly handy for scripts. Try this simple exercise:
$ echo 'echo Hello $USER' > hello.sh$ chmod +x hello.sh
$ ./hello.sh
Hello, Don10. Escalate privileges (sudo)
While administering your system, it may be necessary to act as the super user (also called root). This is where the sudo (or super user do) command comes in. Assuming you're trying to do something that your computer alerts you that only an administrator (or root) user can do, just preface it with the command sudo:
$ touch /etc/os-release && echo "Success"touch: cannot touch '/etc/os-release': Permission denied
$ sudo touch /etc/os-release && echo "Success"
Success11. Shut down (poweroff)
The poweroff command does exactly what it sounds like: it powers your computer down. It requires sudo to succeed.
There are actually many ways to shut down your computer and some variations on the process. For instance, the shutdown command allows you to power down your computer after an arbitrary amount of time, such as 60 seconds:
$ sudo shutdown -h 60Or immediately:
$ sudo shutdown -h nowYou can also restart your computer with sudo shutdown -r now or just reboot.
12. Read the manual (man)The man command could be the most important command of all. It gets you to the documentation for each of the commands on your Linux system. For instance, to read more about mkdir:
$ man mkdirA related command is info, which provides a different set of manuals (as long as they're available) usually written more verbosely than the often terse man pages.
What's your favorite Linux command?There are many more commands on a Linux system—hundreds! What's your favorite command, the one you find yourself using time and time again?
I recommend these commands to anyone who is getting started with Linux.
Image by:Opensource.com
Linux Command line What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 2 Comments Register or Login to post a comment. Greg Pittman | May 24, 2022I generally prefer to use the numbering system for the chmod command. This way I can precisely set permissions with one step. If you list files with ls -l, they will show up like
-rwxrw-r-- example.txtThe first dash shows this is a file, not a directory (you would see a 'd' instead if it were a directory). After that you have 3 groups of rwx permissions (read, write, execute). The first 3 pertain to the owner of the file, the second 3 to his group, and the third to any user.
Greg Pittman | May 24, 2022Numbering involves assigning 4 to r, 2 to w, and 1 to x. So a file such as the one above could be indicated for chmod as 764. If I typed
chmod 664 example.txtls -l would then show it as -rw-rw-r-- example.txt
Also note that the mv command works for directories too.
In reply to I generally prefer to use… by Greg Pittman
7 pieces of Linux advice for beginners
What advice would you give a new Linux user? We asked our community of writers to share their favorite Linux advice.
1. Use Linux resourcesMy brother told me that Linux was like a "software erector set" (that's a dated reference to the old Erector sets that could be purchased in the 1950s and 1960s) which was a helpful metaphor. I was using Windows 3.1 and Windows NT at the time and was trying to build a useful and safe K-12 school district website. This was in 2001 and 2002 and there were very few texts or resources on the web that were helpful. One of the resources recommended was the "Root Users Guide," a very large book that had lots of printed information in it but was tough to decipher and know just how to proceed.
One of the most useful resources for me was an online course that Mandrake Linux maintained. It was a step-by-step explanation of the nuances of using and administering a Linux computer or server. I used that along with a listserv that Red Hat maintained in those days, where you could pose questions and get answers.
More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles 2. Ask the Linux community for helpMy advice is to ask questions, in all of their settings. You can start out with an Internet search, looking for others who have had the same or similar questions (maybe even better questions.) It takes a while to know what to ask and how to ask it.
Once you become more familiar with Linux, check through the various forums out there, to find one or more that you like, and again, before you ask something yourself, look to see if someone else has already had the question and had it answered.
Getting involved in a mail list is also helpful, and eventually, you may find yourself knowledgeable enough to answer some questions yourself. As they say, you learn the most about something by becoming able to answer someone else's questions about it.
Meanwhile, you also become more familiar with using a system that's not a black box that you never understand how something is done except by paying for it.
My advice is to get familiar with help utilities such as man and info. Also, spend as much time as possible at the command line interface and really get used to the fundamental UNIX design. As a matter of fact, one of my favorite books is a UNIX book from the 80s because it really helps in understanding files, directories, devices, basic commands, and more.
The best advice I got was to trust the community with answers and manual pages for detailed information and "how-to" use different options. However, I started off around 2009-ish, there were a lot of tools and resources available, including a project called Linux from Scratch (LFS). This project really taught me a lot about the internals and how to actually build an LFS image.
My advice is to read. Using places like Ask Fedora or the Fedora Matrix chat or other forum type areas. Just read what others are saying, and trying to fix. I learned a lot from just reading what others were struggling with, and then I would try to figure out how the issue was caused.
3. Try dual bootingI started with a dual-boot system in the late 90s (Windows and Linux), and while I wanted to really use Linux, I ended up booting Windows to work in my familiar desktop environment. One of the best pieces of advice was to change the boot order, so every time I wasn't quick enough, I ended up using Linux. ;)
I was challenged by one of my team to do a knowledge swap.
He (our Linux sysadmin) built his website in Joomla! (which our web team specialized in, and he wanted to know more about) and I adopted Linux (having been Windows only to that point.) We dual booted to start with, as I still had a bunch of OS-dependent software I needed to use for the business, but it jump-started my adoption of Linux.
It was really helpful to have each other as an expert to call on while we were each learning our way into the new systems, and quite a challenge to keep going and not give up because he hadn't!
I did have a big sticky note on my monitor saying "anything with rm in the command, ask first" after a rather embarrassing blunder early on. He wrote a command-line cheat sheet (there are dozens online now) for me, which really helped me get familiar with the basics. I also started with the KDE version of Ubuntu, which I found really helpful as a novice used to working with a GUI.
I've used Linux ever since (aside from my work computer) and he's still on Joomla, so it seemed to work for both of us!
4. Back it up for safetyMy advice is to use a distro with an easy and powerful backup app. A new Linux user will touch, edit, destroy and restore configurations. They probably will reach a time when their OS will not boot and losing data is frustrating.
With a backup app, they're always sure that their data is safe.
We all love Linux because it allows us to edit everything, but the dark side of this is that making fatal errors is always an option.
5. Share the Linux you know and useMy advice is to share the Linux you use. I used to believe the hype that there were distributions that were "better" for new users, so when someone asked me to help them with Linux, I'd show them the distro "for new users." Invariably, this resulted in me sitting in front of their computer looking like I had never seen Linux before myself, because something would be just unfamiliar enough to confuse me. Now when someone asks about Linux, I show them how to use what I use. It may not be branded as the "best" Linux for beginners, but it's the distro I know best, so when their problems become mine, I'm able to help solve them (and sometimes I learn something new, myself.)
There was a saying back in the old days, "Do not just use a random Linux distro from a magazine cover. Use the distro your friend is using, so you can ask for help when you need it." Just replace "from a magazine cover" with "off the Internet" and it's still valid :-) I never followed this advice, as I was the only Linux user in a 50km radius. Everyone else was using FreeBSD, IRIX, Solaris, and Windows 3.11 around me. Later I was the one people were asking for Linux help.
6. Keep learning LinuxI was a reseller partner prior to working at Red Hat, and I had a few home health agencies with traveling nurses. They used a quirky package named Carefacts, originally built for DOS, that always got itself out of sync between the traveling laptops and the central database.
The best early advice I heard was to take a hard look at the open source movement. Open source is mainstream in 2022, but it was revolutionary a generation ago when nonconformists bought Red Hat Linux CDs from retailers. Open source turned conventional wisdom on its ear. I learned it was not communism and not cancer, but it scared powerful people.
My company built its first customer firewall in the mid-1990s, based on Windows NT and a product from Altavista. That thing regularly crashed and often corrupted itself. We built a Linux-based firewall for ourselves and it never gave us a problem. And so, we replaced that customer Altavista system with a Linux-based system, and it ran trouble-free for years. I built another customer firewall in late 1999. It took me three weeks to go through a book on packet filtering and get the ipchains commands right. But it was beautiful when I finally finished, and it did everything it was supposed to do. Over the next 15+ years, I built and installed hundreds more, now with iptables; some with bridges or proxy ARP and QOS to support video conferencing, some with IPSEC and OpenVPN tunnels. I got pretty good at it and earned a living managing individual firewalls and a few active/standby pairs, all with Windows systems behind them. I even built a few virtual firewalls.
But progress never stops. By 2022, iptables is obsolete and my firewall days are a fond memory.
The ongoing lesson? Never stop exploring.
7. Enjoy the processBe patient. Linux is a different system than what you are used to, be prepared for a new world of endless possibilities. Enjoy it.
We asked our community of writers for the best advice they got when they first started using Linux.
Image by:Internet Archive Book Images. Modified by Opensource.com. CC BY-SA 4.0
Linux Opensource.com community What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.A hands-on guide to images and containers for developers
Containers and Open Container Initiative (OCI) images are important open source application packaging and delivery technologies made popular by projects like Docker and Kubernetes. The better you understand them, the more able you will be to use them to enhance the consistency and scalability of your projects.
In this article, I will describe this technology in simple terms, highlight the essential aspects of images and containers for a developer to understand, then wrap up by discussing some best practices developers can follow to make their containers portable. I will also walk you through a simple lab that demonstrates building and running images and containers.
What are images?Images are nothing more than a packaging format for software. A great analogy is Java's JAR file or a Python wheel. JAR (or EAR or WAR) files are simply ZIP files with a different extension, and Python wheels are distributed as gzipped tarballs. All of them conform to a standard directory structure internally.
Images are packaged as tar.gz (gzipped tarballs), and they include the software you're building and/or distributing, but this is where the analogy to JARs and wheels ends. For one thing, images package not just your software but all supporting dependencies needed to run your software, up to and including a complete operating system. Whereas wheels and JARs are usually built as dependencies but can be executable, images are almost always built to be executed and more rarely as a dependency.
Knowing the details of what's in the images isn't necessary to understand how to use images or to write and design software for them (if you're interested, read "What is a container image?"). From your perspective, and especially from the perspective of your software, what's important to understand is that the images you create will contain a complete operating system. Because images are packaged as if they're a complete operating system from the perspective of the software you wish to run, they are necessarily much larger than software packaged in a more traditional fashion.
Note that images are immutable. They cannot be changed once they are built. If you modify the software running on the image, you must build an entirely new image and replace the old one.
Linux Containers What are Linux containers? What is Kubernetes? Free online course: Deploy containerized applications eBook: A guide to Kubernetes for SREs and sysadmins Free online course: Running containers with Red Hat technical overview eBook: Storage patterns for Kubernetes TagsWhen images are created, they are created with a unique hash, but they are typically identified with a human-readable name such as ubi, ubi-minimal, openjdk11, and so on. However, there can be different versions of the image for each of their names, and those are typically differentiated by tags. For example, the openjdk11 image might be tagged as jre-11.0.14.1_1-ubi and jre-11.0.14.1_1-ubi-minimal, denoting image builds of the openjdk11 software package version 11.0.14.1_1 installed on a Red Hat ubi and ubi minimal image, respectively.
What are containers?Containers are images that have been realized and executed on a host system. Running a container from an image is a two-step process: create and start. Create takes the image and gives it its own ID and filesystem. Create (as in docker create, for example) can be repeated many times in order to create many instances of a running image, each with its own ID and filesystem. Starting the container will launch an isolated process on the host machine in which the software running inside the container will behave as if it is running in its very own virtual machine. A container is thus an isolated process on the host machine, with its own ID and independent filesystem.
From a software developer's perspective, there are two primary reasons to use containers: consistency and scalability. These are related to each other, and together they allow projects to use one of the most promising innovations to come to software development in recent years, the principle of "Build once, deploy many."
ConsistencyBecause images are immutable and include all of the dependencies needed to run your software from the OS on up, you gain consistency wherever you choose to deploy it. This means whether you launch an image as a container in a development, test, or any number of production environments, the container will run exactly the same way. As a software developer, you won't have to worry about whether any of those environments are running on a different host operating system or version, because the container is running the same operating system every time. That's the benefit of packaging your software along with its complete runtime environment, rather than just your software without the complete set of dependencies needed to run it.
This consistency means that in almost all cases, when an issue is found in one environment (for example, production), you can be confident that you'll be able to reproduce that issue in development or some other environment, so you can confirm the behavior and focus on fixing it. Your project should never get mired in and stumped by the dreaded "But it works on my machine" problem again.
ScalabilityImages contain not only your software but also all the dependencies needed to run your software, including the underlying operating system. This means all processes running inside the container view the container as the host system, the host system is invisible to processes running inside the container, and, from the host system's point of view, the container is just another process it manages. Of course, virtual machines do almost the same thing, which raises a valid question: Why use container technology instead of a virtual machine? The answer lies in both speed and size.
Containers run only the software required to support an independent host without the overhead of having to mimic the hardware. Virtual machines must contain a complete operating system and mimic the underlying hardware. The latter is a very heavyweight solution, which also results in much larger files. Because containers are treated as just another running process from the host system's perspective, they can be spun up in seconds rather than minutes. When your application needs to scale quickly, containers will beat a virtual machine in resources and speed every time. Containers are also easier to scale back down.
Scaling is outside the scope of this article from a functional standpoint, so the lab will not be demonstrating this feature, but it's important to understand the principle in order to understand why container technology represents such a significant advance in the packaging and deployment of software.
Note: While it is possible to run a container that does not include a complete operating system, this is rarely done because the minimal images available are usually an insufficient starting point.
How to find and store imagesLike every other type of software packaging technology, containers need a place where packages can be shared, found, and reused. These are called image registries, analogous to Java Maven and Python wheel repositories or npm registries.
These are a sampling of different image registries available on the internet:
- Docker Hub: The original Docker registry, which hosts many Docker official images used widely among projects worldwide and provides opportunities for individuals to host their own images. One of the organizations that hosts images on Docker Hub is adoptopenjdk; view their repository for examples of images and tags for the openjdk11 project.
- Red Hat Image Registry: Red Hat's official image registry provides images to those with valid Red Hat subscriptions.
- Quay: Red Hat's public image registry hosts many of Red Hat's publicly available images and provides providing opportunities for individuals to host their own images.
There are two utilities whose purpose is to manage images and containers: Docker and Podman. They are available for Windows, Linux, and Mac workstations. From a developer's point of view, they are completely equivalent when executing commands. They can be considered aliases of one another. You can even install a package on many systems that will automatically change Docker into a Podman alias. Wherever Podman is mentioned in this document, Docker can be safely substituted with no change in outcome.
[ Read next: 5 underused Podman features to try now ]
You'll immediately notice these utilities are very similar to Git in that they perform tagging, pushing, and pulling. You will use or refer to this functionality regularly. They should not be confused with Git, however, since Git also manages version control, whereas images are immutable and their management utilities and registry have no concept of change management. If you push two images with the same name and tag to the same repository, the second image will overwrite the first with no way to see or understand what has changed.
SubcommandsThe following are a sampling of Podman and Docker subcommands you will commonly use or refer to:
- build: build an image
- Example: podman build -t org/some-image-repo -f Dockerfile
- image: manage images locally
- Example: podman image rm -a will remove all local images.
- images: list images stored locally
- tag: tag an image
- container: manage containers
- Example: podman container rm -a will remove all stopped local containers.
- run: create and start a container
- also stop and restart
- pull/push: pull/push and image from/to a repository on a registry
Dockerfiles are the source files that define images and are processed with the build subcommand. They will define a parent or base image, copy in or install any extra software you want to have available to run in your image, define any extra metadata to be used during the build and/or runtime, and potentially specify a command to run when a container defined by your image is run. A more detailed description of the anatomy of a Dockerfile and some of the more common commands used in them is in the lab below. A link to the complete Dockerfile reference appears at the end of this article.
Fundamental differences between Docker and PodmanDocker is a daemon in Unix-like systems and a service in Windows. This means it runs in the background all the time, and it runs with root or administrator privileges. Podman is binary. This means it runs only on demand, and can run as an unprivileged user.
This makes Podman more secure and more efficient with system resources (why run all the time if you don't have to?). Running anything with root privileges is, by definition, less secure. When using images on the cloud, the cloud that will host your containers can manage images and containers more securely.
Skopeo and BuildahWhile Docker is a singular utility, Podman has two other related utilities maintained by the Containers organization on GitHub: Skopeo and Buildah. Both provide functionality that Podman and Docker do not, and both are part of the container-tools package group with Podman for installation on the Red Hat family of Linux distributions.
For the most part, builds can be executed through Docker and Podman, but Buildah exists in case more complicated builds of images are required. The details of these more complicated builds are far outside the scope of this article, and you'll rarely, if ever, encounter the need for it, but I include mention of this utility here for completeness.
Skopeo provides two utility functions that Docker does not: the ability to copy images from one registry to another and to delete an image from a remote registry. Again, this functionality is outside the scope of this discussion, but the functionality could eventually be of use to you, especially if you need to write some DevOps scripts.
Dockerfiles labThe following is a very short lab (about 10 minutes) that will teach you how to build images using Dockerfiles and run those images as containers. It will also demonstrate how to externalize your container's configuration to realize the full benefits of container development and "Build once, deploy many."
InstallationThe following lab was created and tested locally running Fedora and in a Red Hat sandbox environment with Podman and Git already installed. I believe you'll get the most out of this lab running it in the Red Hat sandbox environment, but running it locally is perfectly acceptable.
You can also install Docker or Podman on your own workstation and work locally. As a reminder, if you install Docker, podman and docker are completely interchangeable for this lab.
Building Images1. Clone the Git repository from GitHub:
$ git clone https://github.com/hippyod/hello-world-container-lab2. Open the Dockerfile:
$ cd hello-world-container-lab$ vim Dockerfile 1 FROM Docker.io/adoptopenjdk/openjdk11:x86_64-ubi-minimal-jre-11.0.14.1_1
2
3 USER root
4
5 ARG ARG_MESSAGE_WELCOME='Hello, World'
6 ENV MESSAGE_WELCOME=${ARG_MESSAGE_WELCOME}
7
8 ARG JAR_FILE=target/*.jar
9 COPY ${JAR_FILE} app.jar
10
11 USER 1001
12
13 ENTRYPOINT ["java", "-jar", "/app.jar"]
This Dockerfile has the following features:
- The FROM statement (line 1) defines the base (or parent) image this new image will be built from.
- The USER statements (lines 3 and 11) define which user is running during the build and at execution. At first, root is running in the build process. In more complicated Dockerfiles I would need to be root to install any extra software, change file permissions, and so forth, to complete the new image. At the end of the Dockerfile, I switch to the user with UID 1001 so that, whenever the image is realized as a container and executes, the user will not be root, and therefore more secure. I use the UID rather than a username so that the host can recognize which user is running in the container in case the host has enhanced security measures that prevent containers from running as the root user.
- The ARG statements (lines 5 and 8) define variables that can be used during the build process only.
- The ENV statement (line 6) defines an environment variable and value that can be used during the build process but will also be available whenever the image is run as a container. Note how it obtains its value by referencing the variable defined by the previous ARG statement.
- The COPY statement (line 9) copies the JAR file created by the Spring Boot Maven build into the image. For the convenience of users running in the Red Hat sandbox, which doesn't have Java or Maven installed, I have pre-built the JAR file and pushed it to the hello-world-container-lab repo. There is no need to do a Maven build in this lab. (Note: There is also an add command that can be substituted for COPY. Because the add command can have unpredictable behavior, COPY is preferable.)
- Finally, the ENTRYPOINT statement defines the command and arguments that should be executed in the container when the container starts up. If this image ever becomes a base image for a subsequent image definition and a new ENTRYPOINT is defined, it will override this one. (Note: There is also a cmd command that can be substituted for ENTRYPOINT. The difference between the two is irrelevant in this context and outside the scope of this article.)
Type :q and hit Enter to quit the Dockerfile and return to the shell.
3. Build the image:
$ podman build --squash -t test/hello-world -f DockerfileYou should see:
STEP 1: FROM docker.io/adoptopenjdk/openjdk11:x86_64-ubi-minimal-jre-11.0.14.1_1Getting image source signatures
Copying blob d46336f50433 done
Copying blob be961ec68663 done
...
STEP 7/8: USER 1001
STEP 8/8: ENTRYPOINT ["java", "-jar", "/app.jar"]
COMMIT test/hello-world
...
Successfully tagged localhost/test/hello-world:latest
5482c3b153c44ea8502552c6bd7ca285a69070d037156b6627f53293d6b05fd7
In addition to building the image the commands provide the following instructions:
The --squash flag will reduce image size by ensuring that only one layer is added to the base image when the image build completes. Excess layers will inflate the size of the resulting image. FROM, RUN, and COPY/ADD statements add layers, and best practices are to concatenate these statements when possible, for example:
RUN dnf -y --refresh update && \dnf install -y --nodocs podman skopeo buildah && \
dnf clean all
The above RUN statement will not only run each statement to create only a single layer but will also fail the build should any one of them fail.
The -t flag is for naming the image. Because I did not explicitly define a tag for the name (such as test/hello-world:1.0), the image will be tagged as latest by default. I also did not define a registry (such as quay.io/test/hello-world), so the default registry will be localhost.
The -f flag is for explicitly declaring the Dockerfile to be built.
When running the build, Podman will track the downloading of "blobs." These are the image layers your image will be built upon. They are initially pulled from the remote registry, and they will be cached locally to speed up future builds.
Copying blob d46336f50433 doneCopying blob be961ec68663 done
...
Copying blob 744c86b54390 skipped: already exists
Copying blob 1323ffbff4dd skipped: already exists
4. When the build completes, list the image to confirm it was successfully built:
$ podman imagesYou should see:
REPOSITORY TAG IMAGE ID CREATED SIZElocalhost/test/hello-world latest 140c09fc9d1d 7 seconds ago 454 MB
docker.io/adoptopenjdk/openjdk11 x86_64-ubi-minimal-jre-11.0.14.1_1 5b0423ba7bec 22 hours ago 445 MBRunning containers
5. Run the image:
$ podman run test/hello-worldYou should see:
. ____ _ __ _ _/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.5.4)
...
GREETING: Hello, world
GREETING: Hello, world
The output will continue printing "Hello, world" every three seconds until you exit:
crtl-c6. Prove that Java is installed only in the container:
$ java -versionThe Spring Boot application running inside the container requires Java to run, which is why I chose the base image. If you're running in the Red Hat sandbox environment for the lab, this prove sthat Java is installed only in the container, and not on the host:
-bash: java: command not found...Externalize your configurationThe image is now built, but what happens when I want the "Hello, world" message to be different for each environment I deploy the image to? For example, I might want to change it because the environment is for a different phase of development or a different locale. If I change the value in the Dockerfile, I'm required to build a new image to see the message, which breaks one of the most fundamental benefits of containers—"Build once, deploy many." So how do I make my image truly portable so it can be deployed wherever I need it? The answer lies in externalizing the configuration.
7. Run the image with a new, external welcome message:
$ podman run -e 'MESSAGE_WELCOME=Hello, world DIT' test/hello-worldYou should see:
Output:. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.5.4)
...
GREETING: Hello, world DIT
GREETING: Hello, world DIT
Stop using by using crtl-c and adapt the message:
$ podman run -e 'MESSAGE_WELCOME=Hola Mundo' test/hello-world . ____ _ __ _ _/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.5.4)
...
GREETING: Hola Mundo
GREETING: Hola Mundo
The -e flag defines an environment variable and value to inject into the container at startup. As you can see, even if the variable was built into the original image (the ENV MESSAGE_WELCOME=${ARG_MESSAGE_WELCOME} statement in your Dockerfile), it will be overridden. You've now externalized data that needed to change based on where it was to be deployed (for example, in a DIT environment or for Spanish speakers) and thus made your images portable.
8. Run the image with a new message defined in a file:
$ echo 'Hello, world from a file' > greetings.txt$ podman run -v "$(pwd):/mnt/data:Z" \
-e 'MESSAGE_FILE=/mnt/data/greetings.txt' test/hello-world
In this case you should see:
. ____ _ __ _ _/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.5.4)
...
GREETING: Hello, world from a file
GREETING: Hello, world from a file
Repeat until you hit crtl-c to stop
The -e flag in this case defines a path to the file at /mnt/data/greetings.txt that was mounted from the host's local file system with the -v flag at $(pwd)/greetings.txt (pwd is a bash utility that outputs the absolute path of the current directory, which in your case should be the hello-world-container-lab). You've now externalized data that needed to change based on where it was to be deployed, but this time your data was defined in an external file you mounted into the container. Environment variable settings are OK for a limited number of settings, but when you have several settings to apply, a file is a more efficient way of injecting the values into your containers.
Note: The :Z flag at the end of the volume definition above is for systems using SELinux. SELinux manages security on many Linux distributions, and the flag allows the container access to the directory. Without the flag, SELinux would prevent the reading of the file, and an exception would be thrown in the container. Try running the command above again after removing the :Z to see a demonstration.
This concludes the lab.
Developing for containers: externalize the configuration"Build once, deploy many" works because the immutable containers running in different environments don't have to worry about differences in the hardware or software required to support your particular software project. This principle makes software development, debugging, deployment, and ongoing maintenance much faster and easier. It also isn't perfect, and some minor changes have to be made in how you code to make your container truly portable.
The most important design principle when writing software for containerization is deciding what to externalize. These decisions ultimately make your images portable so they can fully realize the "Build once, deploy many" paradigm. Although this may seem complicated, there are some easy-to-remember factors to consider when deciding whether the configuration data should be injectable into your running container:
- Is the data environment-specific? This includes any data that needs to be configured based on where the container is running, whether the environment is a production, non-production, or development environment. Data of this sort includes internationalization configuration, datastore information, and the specific testing profile(s) you want your application to run under.
- Is the data release independent? Data of this sort can run the gamut from feature flags to internationalization files to log levels—basically, any data you might want or need to change between releases without a build and new deployment.
- Is the data a secret? Credentials should never be hard coded or stored in an image. Credentials typically need to be refreshed on schedules that don't match release schedules, and embedding a secret in an image stored in an image registry is a security risk.
The best practice is to choose where your configuration data should be externalized (that is, in an environment variable or a file) and only externalize those pieces that meet the above criteria. If it doesn't meet the above criteria, it is best to leave it as part of the immutable image. Following these guidelines will make your images truly portable and keep your external configuration reasonably sized and manageable.
[ Free online course: Containers, Kubernetes and Red Hat OpenShift Technical Overview ]
SummaryThis article introduces four key ideas for software developers new to images and containers:
- Images are immutable binaries: Images are a means of packaging software for later reuse or deployment.
- Containers are isolated processes: When they are created, containers are a runtime instantiation of an image. When containers are started, they become processes in memory on a host machine, which is much lighter and faster than a virtual machine. For the most part, developers only need to know the latter, but understanding the former is helpful.
- "Build once, deploy many": This principle is what makes container technology so useful. Images and containers provide consistency in deployments and independence from the host machine, allowing you to deploy with confidence across many different environments. Containers are also easily scalable because of this principle.
- Externalize the configuration: If your image has configuration data that is environment-specific, release-independent, or secret, consider making that data external to the image and containers. You can inject this data into your running image by injecting an environment variable or mounting an external file into the container.
A brief history of containers: from the 1970s till now
A practical introduction to container terminology
Understand the key concepts behind images and containers. Then try a lab that demonstrates building and running images and containers.
Image by:Lucarelli via Wikimedia Commons. CC-BY-SA 3.0
Containers What to read next Experiment with containers and pods on your own computer A beginner's guide to Kubernetes container orchestration This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.How to rename a branch, delete a branch, and find the author of a branch in Git
One of Git's primary strengths is its ability to "fork" work into different branches.
If you're the only person using a repository, the benefits are modest, but once you start working with many other contributors, branching is essential. Git's branching mechanism allows multiple people to work on a project, and even on the same file, at the same time. Users can introduce different features, independent of one another, and then merge the changes back to a main branch later. A branch created specifically for one purpose, such as adding a new feature or fixing a known bug, is sometimes called a topic branch.
Once you start working with branches, it's helpful to know how to manage them. Here are the most common tasks developers do with Git branches in the real world.
More on Git What is Git? Git cheat sheet Markdown cheat sheet New Git articles Rename a branch using GitRenaming a topic branch is useful if you have named a branch incorrectly or you want to use the same branch to switch between different bugs or tasks after merging the content into the main branch.
Rename a local branch1. Rename the local branch:
$ git branch -m <old_branch_name> <new_branch_name>Of course, this only renames your copy of the branch. If the branch exists on the remote Git server, continue to the next steps.
2. Push the new branch to create a new remote branch:
$ git push origin <new_branch_name>3. Delete the old remote branch:
$ git push origin -d -f <old_branch_name>Rename the current branchWhen the branch you want to rename is your current branch, you don't need to specify the existing branch name.
1. Rename the current branch:
$ git branch -m <new_branch_name>2. Push the new branch to create a new remote branch:
$ git push origin <new_branch_name>3. Delete the old remote branch:
$ git push origin -d -f <old_branch_name>Delete local and remote branches using GitAs part of good repository hygiene, it's often recommended that you delete a branch after ensuring you have merged the content into the main branch.
Delete a local branchDeleting a local branch only deletes the copy of that branch that exists on your system. If the branch has already been pushed to the remote repository, it remains available to everyone working with the repo.
1. Checkout the central branch of your repository (such as main or master):
$ git checkout <central_branch_name>2. List all the branches (local as well as remote):
$ git branch -a3. Delete the local branch:
$ git branch -d <name_of_the_branch>To remove all your local topic branches and retain only the main branch:
$ git branch | grep -v main | xargs git branch -dDelete a remote branchDeleting a remote branch only deletes the copy of that branch that exists on the remote server. Should you decide that you didn't want to delete the branch after all, you can re-push it to the remote, such as GitHub, as long as you still have your local copy.
1. Checkout the central branch of your repository (usually main or master):
$ git checkout <central_branch_name>2. List all branches (local as well as remote):
$ git branch -a3. Delete the remote branch:
$ git push origin -d <name_of_the_branch>Find the author of a remote topic branch using GitIf you are the repository manager, you might need to do this so you can inform the author of an unused branch that it should be deleted.
1. Checkout the central branch of your repository (such as main or master):
$ git checkout <central_branch_name>2. Delete branch references to remote branches that do not exist:
$ git remote prune origin3. List the author of all the remote topic branches in the repository, using the --format option along with special selectors (in this example, %(authorname) and %(refname) for author and branch name) to print just the information you want:
$ git for-each-ref --sort=authordate --format='%(authorname) %(refname)' refs/remotesExample output:
tux refs/remotes/origin/devagil refs/remotes/origin/main
You can add further formatting, including color coding and string manipulation, for easier readability:
$ git for-each-ref --sort=authordate \--format='%(color:cyan)%(authordate:format:%m/%d/%Y %I:%M %p)%(align:25,left)%(color:yellow) %(authorname)%(end)%(color:reset)%(refname:strip=3)' \
refs/remotes
Example output:
01/16/2019 03:18 PM tux dev05/15/2022 10:35 PM agil main
You can use grep to get the author of a specific remote topic branch:
$ git for-each-ref --sort=authordate \--format='%(authorname) %(refname)' \
refs/remotes | grep <topic_branch_name>Get good at branching
There are nuances to how Git branching works depending on the point at which you want to fork the code base, how the repository maintainer manages branches, squashing, rebasing, and so on. Here are three articles for further reading on this topic:
- Explaining Git branches with a LEGO analogy, by Seth Kenlon
- My guide to using the Git push command safely, by Noaa Barki
- A guide to Git branching, by Kedar Vijay Kulkarni
Become an expert at the most common Git tasks for managing local and remote branches.
Image by:Image by Erik Fitzpatrick. CC BY-SA 4.0
Git What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.A programmer's guide to GNU C Compiler
C is a well-known programming language, popular with experienced and new programmers alike. Source code written in C uses standard English terms, so it's considered human-readable. However, computers only understand binary code. To convert code into machine language, you use a tool called a compiler.
A very common compiler is GCC (GNU C Compiler). The compilation process involves several intermediate steps and adjacent tools.
Install GCCTo confirm whether GCC is already installed on your system, use the gcc command:
$ gcc --versionIf necessary, install GCC using your packaging manager. On Fedora-based systems, use dnf:
$ sudo dnf install gcc libgccOn Debian-based systems, use apt:
$ sudo apt install build-essentialAfter installation, if you want to check where GCC is installed, then use:
$ whereis gccSimple C program using GCCHere's a simple C program to demonstrate how to compile code using GCC. Open your favorite text editor and paste in this code:
// hellogcc.c#include
int main() {
printf("Hello, GCC!\n");
return 0;
}
Save the file as hellogcc.c and then compile it:
$ lshellogcc.c
$ gcc hellogcc.c
$ ls -1
a.out
hellogcc.c
As you can see, a.out is the default executable generated as a result of compilation. To see the output of your newly-compiled application, just run it as you would any local binary:
$ ./a.outHello, GCC!Name the output file
The filename a.out isn't very descriptive, so if you want to give a specific name to your executable file, you can use the -o option:
$ gcc -o hellogcc hellogcc.c$ ls
a.out hellogcc hellogcc.c
$ ./hellogcc
Hello, GCC!
This option is useful when developing a large application that needs to compile multiple C source files.
Programming and development Red Hat Developers Blog Programming cheat sheets Try for free: Red Hat Learning Subscription eBook: An introduction to programming with Bash Bash shell scripting cheat sheet eBook: Modernizing Enterprise Java Intermediate steps in GCC compilationThere are actually four steps to compiling, even though GCC performs them automatically in simple use-cases.
- Pre-Processing: The GNU C Preprocessor (cpp) parses the headers (#include statements), expands macros (#define statements), and generates an intermediate file such as hellogcc.i with expanded source code.
- Compilation: During this stage, the compiler converts pre-processed source code into assembly code for a specific CPU architecture. The resulting assembly file is named with a .s extension, such as hellogcc.s in this example.
- Assembly: The assembler (as) converts the assembly code into machine code in an object file, such as hellogcc.o.
- Linking: The linker (ld) links the object code with the library code to produce an executable file, such as hellogcc.
When running GCC, use the -v option to see each step in detail.
$ gcc -v -o hellogcc hellogcc.c Image by:(Jayashree Huttanagoudar, CC BY-SA 4.0)
Manually compile codeIt can be useful to experience each step of compilation because, under some circumstances, you don't need GCC to go through all the steps.
First, delete the files generated by GCC in the current folder, except the source file.
$ rm a.out hellogcc.o$ ls
hellogcc.cPre-processor
First, start the pre-processor, redirecting its output to hellogcc.i:
$ cpp hellogcc.c > hellogcc.i$ ls
hellogcc.c hellogcc.i
Take a look at the output file and notice how the pre-processor has included the headers and expanded the macros.
CompilerNow you can compile the code into assembly. Use the -S option to set GCC just to produce assembly code.
$ gcc -S hellogcc.i$ ls
hellogcc.c hellogcc.i hellogcc.s
$ cat hellogcc.s
Take a look at the assembly code to see what's been generated.
AssemblyUse the assembly code you've just generated to create an object file:
$ as -o hellogcc.o hellogcc.s$ ls
hellogcc.c hellogcc.i hellogcc.o hellogcc.sLinking
To produce an executable file, you must link the object file to the libraries it depends on. This isn't quite as easy as the previous steps, but it's educational:
$ ld -o hellogcc hellogcc.old: warning: cannot find entry symbol _start; defaulting to 0000000000401000
ld: hellogcc.o: in function `main`:
hellogcc.c:(.text+0xa): undefined reference to `puts'
An error referencing an undefined puts occurs after the linker is done looking at the libc.so library. You must find suitable linker options to link the required libraries to resolve this. This is no small feat, and it's dependent on how your system is laid out.
When linking, you must link code to core runtime (CRT) objects, a set of subroutines that help binary executables launch. The linker also needs to know where to find important system libraries, including libc and libgcc, notably within special start and end instructions. These instructions can be delimited by the --start-group and --end-group options or using paths to crtbegin.o and crtend.o.
This example uses paths as they appear on a RHEL 8 install, so you may need to adapt the paths depending on your system.
$ ld -dynamic-linker \/lib64/ld-linux-x86-64.so.2 \
-o hello \
/usr/lib64/crt1.o /usr/lib64/crti.o \
--start-group \
-L/usr/lib/gcc/x86_64-redhat-linux/8 \
-L/usr/lib64 -L/lib64 hello.o \
-lgcc \
--as-needed -lgcc_s \
--no-as-needed -lc -lgcc \
--end-group
/usr/lib64/crtn.o
The same linker procedure on Slackware uses a different set of paths, but you can see the similarity in the process:
$ ld -static -o hello \-L/usr/lib64/gcc/x86_64-slackware-linux/11.2.0/ \
/usr/lib64/crt1.o /usr/lib64/crti.o \
hello.o /usr/lib64/crtn.o \
--start-group -lc -lgcc -lgcc_eh \
--end-group
Now run the resulting executable:
$ ./helloHello, GCC!Some helpful utilities
Below are a few utilities that help examine the file type, symbol table, and the libraries linked with the executable.
Use the file utility to determine the type of file:
$ file hellogcc.chellogcc.c: C source, ASCII text
$ file hellogcc.o
hellogcc.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
$ file hellogcc
hellogcc: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=bb76b241d7d00871806e9fa5e814fee276d5bd1a, for GNU/Linux 3.2.0, not stripped
The use the nm utility to list symbol tables for object files:
$ nm hellogcc.o0000000000000000 T main
U puts
Use the ldd utility to list dynamic link libraries:
$ ldd hellogcclinux-vdso.so.1 (0x00007ffe3bdd7000)
libc.so.6 => /lib64/libc.so.6 (0x00007f223395e000)
/lib64/ld-linux-x86-64.so.2 (0x00007f2233b7e000)Wrap up
In this article, you learned the various intermediate steps in GCC compilation and the utilities to examine the file type, symbol table, and libraries linked with an executable. The next time you use GCC, you'll understand the steps it takes to produce a binary file for you, and when something goes wrong, you know how to step through the process to resolve problems.
Get a behind-the-scenes look at the steps it takes to produce a binary file so that when something goes wrong, you know how to step through the process to resolve problems.
Image by:Opensource.com
Programming What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Use this open source screen reader on Windows
Screen readers are a specialized area of assistive technology software that reads and then speaks the content of a computer screen. People with no sight at all are just a fraction of those with visual impairments, and screen reader software can help all groups. Screen readers are mostly specific to an operating system, and they're used by people with visual impairments and accessibility trainers, as well as developers and accessibility consultants wanting to test how accessible a website or application is.
More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources How to use the NVDA screen readerThe WebAIM screen reader user surveys began in 2009 and ran to 2021. In the first survey, the most common screen reader used was JAWS at 74%. It is a commercial product for Microsoft Windows, and the long-time market leader. NVDA, then a relatively new open source screen reader for Windows came in at just 8%. Fast forward to 2021 and JAWS comes in with 53.7% with NVDA at 30.7%.
You can download the latest release of NVDA from the NVAccess website. Why do I use NVDA and recommend it to my MS Windows using clients? Well, it is open source, fast, powerful, easy to install, supports a wide variety of languages, can be run as a portable application, has a large user base, and there is a regular release cycle for new versions.
NVDA has been translated into fifty-five languages and is used in one-hundred and seventy-five different countries. There is also an active developer community with their own Community Add-ons website. Any add-ons you choose to install will depend on your needs and there are a lot to choose from, including extensions for common video conferencing platforms.
Like all screen readers, there are a lot of key combinations to learn with NVDA. Using any screen reader proficiently takes training and practice.
Image by:(Peter Cheer, CC BY-SA 4.0)
Teaching NVDA to people familiar with computers and who have keyboard skills is not too difficult. Teaching basic computer skills (without the mouse, touch pad, and keyboard skills) and working with NVDA to a complete beginner is far more of a challenge. Individual learning styles and preferences differ. In addition, people may not need to learn how to do everything if all that they want to do is browse the web and use email. A good source of links to NVDA tutorials and resources is Accessibility Central.
It becomes easier once you have mastered operating NVDA with keyboard commands, but there is also a menu-driven system for many configuration tasks.
Image by:(Peter Cheer, CC BY-SA 4.0)
Test for accessibilityThe inaccessibility of some websites to screen reader users has been a problem for many years, and still is despite disability equality legislation like the Americans with Disabilities Act (ADA). An excellent use for NVDA in the sighted community is for website accessibility testing. NVDA is free to download, and by running a portable version, website developers don't even need to install it. Run NVDA, turn off your monitor or close your eyes, and see how well you can navigate a website or application.
NVDA can also be used for testing when working through the (often ignored) task of properly tagging a PDF document for accessibility.
There are several guides that concentrate on using NVDA for accessibility testing. I can recommend Testing Web Pages with NVDA and Using NVDA to Evaluate Web Accessibility.
In honor of Global Accessibility Awareness Day, learn about the NVDA open source screen reader and how you can get involved to improve accessibility for all web users.
Image by:Opensource.com
Accessibility Windows What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Near zero marginal cost societies and the impact on why we work
I have read Jeremy Rifkin's book The Zero Marginal Cost Society: The Internet of Things, the Collaborative Commons, and the Eclipse of Capitalism, which has a strong connection to open organization principles, particularly community building. Rifkin also writes about the future of green energy generation and energy use in logistics. This is the second of three articles in this series. In my previous article, I examined the Collaborative Commons. In this article, I look at its impact on energy production and supply.
Within the next 25 years, Rifkin believes most of our energy for home heating, running appliances, powering businesses, driving vehicles, and operating the whole economy will be nearly free with on-site power solar, wind and geothermal energy generation. This is starting already, through both individual and micropower plants. The payback is around two to eight years.
What would be the best organizational structure to manage nearly free green energy? Furthermore, through an intelligent communication and energy system, an organization could generate business anywhere in the world, and share energy across a continental energy internet. On top of that, it could produce and sell goods at a fraction charged by global manufacturing giants.
The Internet of Things is on the wayAccording to Rifkin, the Internet of Things (IoT) will connect every machine, business, residence, and vehicle in an intelligent network that consists of not just a communications internet like now, but in the future an energy internet, and a logistics internet. They will all be embedded in a single operating system. Energy use will be completely monitored. Rifkin believes that within 10 years, many smart energy meters will be in use (by 2025). All this investment will reduce at least 10% of the waste in the current industrial system.
All this will be possible with the reduction of costs of sensors and actuators embedded in devices. Radio-frequency identification (RFID) chip prices have fallen by 40% in around 2012-2013. Micro-electromechanical system (MEMS), including gyroscopes, accelerometers, and pressure sensors, have also dropped by 80-90% in price over the past five years (up to 2015 when this book was published).
This will increase device connections, to as many as 1,000 connections on one person's devices, appliances, and facilities. This connection is what young people love, total inclusion in a global virtual public square to share everything. They thrive on transparency, collaboration, and inclusivity with care taken to an appropriate level of privacy. So, you can see, the time is right for the growth of a Collaborative Commons in society.
Open Organization resources Download resources Join the community What is an open organization? How open is your organization? Exponential curveQuestion: Would you accept US $1,000,000 today or US $1.00 that doubled every day for 30 days? (In 30 days it will be US $536,870,912. That is 500 times more). In 31 days, over one billion. Exponential growth is deceptively fast. That is how fast costs are coming down according to Rifkin. This will turn the entire fossil fuel industry investments into stranded assets. We should be planning for the Collaborative Commons using all open organization principles now, as the situation will be ideal for them very soon.
Next, a free energy internetAt this point in time there is free information for learning if you look for it. The next step is free energy (solar, wind, geothermal, biomass, hydro). After initial investments (research, development, deployment), Rifkin forecasts that unit costs will rapidly come down. The information internet and near zero-cost renewables will merge into the energy internet, powering homes, offices, and factories. Ideally, there will be energy that's loaded into buildings and partially stored in the form of hydrogen, distributed over a green-electricity internet, and connected to plug-in, zero-emission transportation. The development of renewable energy establishes a five pillar mechanism that will allow billions of people to share energy at near zero marginal cost in the IoT world
Solar energyIf you start collecting energy from the sun, facilities only need to obtain 00.1% of the sun's energy that reaches the Earth. That would provide six times the energy we now use globally.
SunPower Corporation is one company doing that. It supports making homes energy producers. The price of solar photovoltaic (PV) cells tends to drop by 20% for every doubling of industry capacity. Solar panels are increasing, and their ability to capture more energy per panel is increasing. Expect to see the development of thinner solar panels, and paper thin solar sheets. Eventually there will be solar energy paint and solar energy windows in the future.
When too much energy is generated, it must be sold elsewhere or stored in batteries or used to produce hydrogen. This technology is coming to the market and will dominate it very soon. With these technologies alone, electricity is on the way to have zero marginal cost.
Wind power generationWind power has been growing exponentially since the 1990s, and is now nearing fossil fuel and nuclear power generation levels (as of 2015). With the lowering costs of windmill production, installation, and maintenance, wind power is doubling every 2-½ years. With the increase of solar and wind energy sources, governments do not need to subsidize them with tariffs any longer.
Energy Watch Group is tracking this. According to Rifkin geothermal energy, biomass, and wave and tidal power will likely reach their own exponential takeoff stage within the next decade. He believes that all this will happen in the first half of the twenty-first century. If this capacity doubles eight more times, by 2028, 80% of all energy generation will be from these renewables.
The collaborative age will soon be hereWith all the above developments, society's working and living environment are changing. According to the collaborative age: "This means ensuring that people can collaborate on tasks without friction. That humans and machines work seamlessly together. And automation — machine to machine collaboration — will be crucial. Those businesses that get these elements right will be able to boost employee satisfaction and attract the best talent. They will reduce costs by automating mundane tasks and by requiring a smaller office footprint. The social impact-focused organizations that Laura Hilliger and Heather Leson write about how to take advantage of this new age.
Making the transitionIt sounds good, but how do businesses transition from the information age to the collaboration age? One area will be in decentralized production through 3D printing technology (additive manufacturing, not cutting away and creating waste).
Instead of shipping goods, in the future, software will be shipped for many items to be manufactured locally, avoiding all shipping costs and making manufacturing become on-site near where the market need is. Locally, newly developed molten plastics, molten metal, or other feedstock inside a printer will be used for fabrication. This will give one 3D printer the ability to produce tens of thousands of products (like jewelry, airplane parts, and human prostheses).
Centralized manufacturing vs local production which Rifkin projects will comeRifkin believes lower marketing costs are possible by using the IoT economy and using global internet marketing sites at almost zero marginal cost.
-
There is little human involvement in the local 3D process (giving the name "infofacture" rather than manufacture. They ship the information required for local manufacturing, like downloading music today. It is just code that you receive.
-
The code for 3D printing is open source, so people can learn and improve designs, becoming further prosumers in a wide range of items (no intellectual-property protection barriers). This will lead to exponential growth over the next several decades, offering more complicated products at lower prices and near zero marginal cost.
-
There is great waste with subtraction processes (current manufacturing processes) producing great waste with each process. (1/10 the materials required. This material could be developed from subatomic particles that are available anywhere in the local environment, like recycled glass, fabrics, ceramics, and stainless steel. Composite-fiber concrete could be extruded form-free and be strong enough for building construction walls [probably available in two decades].)
-
3D printing processes have fewer moving parts and less spare parts. Therefore, expensive retooling and changeover delays will be less extensive.
-
Materials will be more durable, recyclable, and less polluting.
-
Local distributed production, through IoT, will spread globally at an exponential rate with little shipping cost and less use of energy.
Rifkin cites Etsy as an example of this model. You can find things you are interested in, and have them produced locally using their network. They sell the code, and you can have it supplied in your area.
Rifkin feels that in the future, small and medium sized 3D businesses, infofacturing more sophisticated products, will likely cluster in local technology parks to establish an optimum lateral scale (another form of community development). Here are current examples:
-
RepRap: This is a manufacturing machine that can produce itself and all its parts.
-
Thingiverse The largest 3D printing community. They share under the General Public Licenses (GPL) and Creative Commons Licenses.
-
Fab Lab: Open source peer-to-peer learning in manufacturing. It is being provided to local, distant communities in developing countries.
-
3D Printed automobiles (Urbee vehicle) is already being tested.
-
KOR EcoLogic has an urban electric vehicle.
Here are the principles that these ecosystems follow:
-
They use open source shared software.
-
They promote a collaborative learning culture.
-
They believe that it will build a self-sufficient community.
-
They are committed to sustainable production practices.
When technology replaces workers, capital investments replace labor investments. In 2007, companies used 6 times more computers and software than 20 years before, doubling the amount of capital used per hour of employee work. The robot workforce is on the rise. China, India, Mexico, and other emerging nations are learning that the cheapest workers in the world are not as cheap, efficient, or productive as the information technology, robotics, and artificial intelligence that replaces them.
Going back to Rifkin, the first industrial revolution ended slave and serf labor. The second industrial revolution will dramatically shrink agricultural and craft labor. Rifkin believes the third industrial revolution will be a decline in mass wage labor in the manufacturing, service industries, and salaried professional labor in large parts of the knowledge sector.
Rifkin believes that an abundance, zero marginal cost economy, will change our notion of economic processes. He thinks the old paradigm of owners and workers, sellers and consumers will break down. Consumers will start producing for themselves (and a few others), eliminating their distinction. Prosumers will increasingly be able to produce, consume, and share their own goods and services with one another on the Collaborative Commons at diminishing marginal costs approaching zero, bringing to the fore new ways of organizing economic life beyond the traditional capitalist market mode.
Rifkin forecasts that in the future, greater importance will be placed on the Collaborative Commons and be as important as hard work was in the market economy (one's ability to cooperate and collaborate as opposed to just working hard). The amassing of social capital will become as valued as the accumulation of market capital. Attachment to community and the search for transcendence and meaning comes to define the measure of one's material wealth. All the open organization principles we write about will be exponentially more important in the future.
The IoT will free human beings from the capitalist market economy to pursue nonmaterial shared interests on the collaborative commons. Many — but not all — of our basic material needs will be met for nearly free in a near zero marginal cost society. It will be abundance over scarcity.
Prosumer and the entry of the smart economyRifkin writes that as capitalist economies step aside in some global commodities, in the collaborative commons, sellers and buyers will give way to prosumers, property rights will make room for open source sharing, ownership will be less important than access, markets will be superseded by networks, and the marginal cost of supplying information, generating energy, manufacturing products, and teaching students will become nearly zero.
Internet of energy is on the wayFinancing of the IoT will not come from wealthy capitalists and corporate shareholders, but from hundreds of millions of consumers and taxpayers. No one owns the internet. It is only in operation because a set of agreed-upon protocols were established that allows computer networks to communicate with each other. It is a virtual public square for all who pay for a connection to use it. Next comes distributed renewable energies that will be distributed in the same way. Supported by feed-in tariffs and other fund-raising methods, governments will pay for the initial research, but after that fixed investment, the public will be able to connect and use it freely. Once underway, governmental centralized operations will move to distributed ownership. The Electric Power Research Institute (EPRI), is studying how to build a national energy internet over the next 20 years.
This is not just supplying electricity. Every device in every building will be equipped with sensors and software that connect to the IoT, feeding real-time information on electricity use to both the on-site prosumer and the rest of the network. The entire network will know how much electricity is being used by every appliance at any moment — thermostats, washing machines, dishwashers, televisions, hair dryers, toasters, ovens, and refrigerators.
This is not happening in the future, but now. It is not just being considered but being done now. Intwine Energy can supply the above process now. The issue is getting it into the greater global population. A group of young social entrepreneurs are now using social media to mobilize their peers to create, operate and use the energy internet.
A new world of abundance beyond our needsRifkin thinks society has to start envisioning an entire different living environment. Imagine a world in which you can just give away things you once had to pay for, or had to sell at a profit. No one charges us for each internet connected phone call. He believes these give-away goods need not be covered by governments, like telecommunication, roads, bridges, public schools or hospitals. They need not be considered totally private property to be sold and bought, either. These goods have to be supplied in communities with rules, responsibilities and joint benefits (information, energy, local production, and online education). Not governed by the markets or governments, but by networked commons because of the tragedy of the commons. It governs and enforces distributed, peer-to-peer, laterally scaled economic activities.
Rifkin feels the Collaborative Commons as a governing body is extremely important. This is where local (project) leadership comes in. The goals, processes, tasks and responsibilities must be successfully executed, after they have been decided and agreed on. Furthermore, "social capital" is a major factor. It must be widely introduced and deepened in quality. Community exchange, interaction and contribution is far more important than selling to distant capital markets. If that is the case, our open organization leadership will be extremely important.
The public square versus private ownership"The public square at — least before the Internet, is where we communicate, socialize, revel in each other's company, establish communal bonds, and create social capital and trust. These are all indispensable elements for a nurturing community." Historically, Japanese villages were built like that to survive natural, economic and political disasters like earthquakes and typhoons. They put common interests over self-interests This is what the open organization principle of community is all about.
The right to be included, to have access to one another, which is the right to participate together, is a fundamental right of all. Private property, the right to enclose, own, and exclude is merely a qualified deviation from the norm. For some reason, having massive private property rights have gained in importance in more recent modern times. This will all be reversed in the years ahead according to Rifkin.
Rifkin writes that the world will move to these commons:
-
Public square commons
-
Land commons
-
Virtual commons
-
Knowledge commons (languages, cultures, human knowledge and wisdom)
-
Energy Commons
-
Electromagnetic spectrum commons
-
Ocean commons
-
Fresh water commons
-
Atmosphere commons
-
Biosphere commons.
The past 200 years of capitalism, the enclosed, privatized, and commodification of the market must be put under review. How would they be most effective in a transparent, non-hierarchical and collaborative culture? It comes down to two views, the capitalist (I own it. It is mine, and you can't use it) and the collaborationist (This is for everyone to use, and there are rules and guidelines to use it, so everyone can get their fair share). Today's cooperatives are good at this, like the International Co-operative Alliance (ICA). Cooperatives have to generate motivation for the greater community good, and that motivation must be greater than any profit motive. That is their challenge but this not new, as one in seven people on the earth are in some kind of cooperative now.
As I've presented in this article, the IoT will become our working and living environment. Also, energy costs are projected to go to near zero. With those changes, community collaboration and cooperation will become ever more important over hard work. In the last part of this series I will look at Collaborative Commons in logistics and other economic activity.
As the IoT becomes our working and living environment, energy costs will come closer to zero and community collaboration will be critical.
Image by:Opensource.com
The Open Organization What to read next How collaborative commons and open organization principles align This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.A guide to Pipy, a programmable network proxy for cloud
Pipy is an open source, cloud-native, network stream processor. It is modular by design and can create a high-performance network proxy. It's written in C++ and is built on top of the Asio asynchronous I/O library. Pipy is ideal for a variety of use cases ranging from edge routers, load balancers, proxy solutions, API gateways, static HTTP servers, service mesh sidecars, and more.
Pipy also comes with built-in JavaScript support through PipyJS. PipyJS is highly customizable and predictable in performance, with no garbage collection overhead. Currently, PipyJS is part of the Pipy code base, but it has no dependency on it and in the future it may be moved to a standalone package.
Pipy quick start guideYou can run the production version of Pipy using Podman or Docker with one of the tutorial scripts provided on the official Pipy Git repository. The Pipy container image can be configured with a few environment variables:
-
PIPY_CONFIG_FILE= sets the location of the Pipy configuration file.
-
PIPY_SPAWN=n sets the number of Pipy instances you want to start, where n is the number of instances. This is a zero-based index, so 0 represents 1 instance. For example, use PIPY_SPAWN=3 for 4 instances.
Start the Pipy server with this example script:
$ docker run --rm -e PIPY_CONFIG_FILE=\https://raw.githubusercontent.com/flomesh-io/pipy/main/tutorial/01-hello \
-e PIPY_SPAWN=1 -p 8080:8080 flomesh/pipy-pjs:latest
You may notice that instead of a local file, this code provides a link to a remote Pipy script through the environment variable PIPY_CONFIG_FILE. Pipy is smart enough to handle that.
For your reference, here are the contents of the file tutorial/01-hello/hello.js:
pipy().listen(8080)
.serveHTTP(
new Message('Hi, there!\n')
)
This simple script defines one Port pipeline, which listens on port 8080 and returns "Hi, there!" for each HTTP request received on the listening port.
As you've exposed local port 8080 with the docker run command, you can proceed with a test on the same port:
$ curl http://localhost:8080Executing the above command displays Hi, there! on the console.
For learning, development, or debugging purposes it's recommended to proceed with the local installation (either build Pipy from sources or download a release for your OS) of Pipy, as it comes with an admin web console along with documentation and tutorials.
Once installed locally, running pipy without any arguments starts the admin console on port 6060, but it can be configured to listen on the different port with the --admin-port option.
Image by:(Ali Naqvi, CC BY-SA 40)
To build Pipy from source, or to install a precompiled binary for your operating system, refer to README.md on the Pipy Git repository.
Explore the open source cloud Free online course: Developing cloud-native applications with microservices eBook: Modernize your IT with managed cloud services Try for 60 days: Red Hat OpenShift Dedicated Free online course: Containers, Kubernetes and Red Hat OpenShift What is Kubernetes? Understanding edge computing Latest articles for IT architects Running Pipy in a terminalTo start a Pipy proxy, run Pipy with a PipyJS script file, for example, the script in tutorial/01-hello/hello.js if you need a simple echo server that responds with the same message body received with every incoming request:
$ pipy tutorial/01-hello/hello.jsAlternatively, while developing and debugging, one can start Pipy with a builtin web UI:
$ pipy tutorial/01-hello/hello.js --admin-port=6060To see all command-line options, use the --help flag:
$ pipy --helpPipy is a stream processorPipy operates on network streams using an event-driven pipeline where it consumes the input stream, performs user-provided transformations, and outputs the stream. A pipy data stream takes raw data and abstracts it into an event. An event can belong to one of four categories:
- Data: Network streams are composed of data bytes and come in chunks. Pipy abstracts out chunks into a Data event.
-
MessageStart, MessageEnd, StreamEnd: These three non-data events work as markers, giving the raw byte streams high-level semantics for business logic to rely on.
The internal workings of Pipy are similar to Unix Pipelines but unlike Unix pipelines, which deal with discreet bytes, Pipy deals with streams of events.
Pipy processes incoming streams through a chain of filters, where each filter deals with general concerns like request logging, authentication, SSL offloading, request forwarding, and so on. Each filter reads from its input and writes to its output, with the output of one filter connected to the input of the next.
PipelinesA chain of filters is called a pipeline and Pipy categorizes pipelines in 3 different categories according to their input sources.
-
Port pipeline: Reads in Data events from a network port, processes them, and then writes the result back to the same port. This is the most commonly used request and response model. For instance, when Pipy works like an HTTP server, the input to a Port pipeline is an HTTP request from the clients, and the output from the pipeline would be an HTTP response sent back to clients.
-
Timer pipeline: Gets a pair of MessageStart and MessageEnd events as its input periodically. Useful when Cron job-like functionality is required.
-
Sub-pipeline: Works in conjunction with a join filter, such as link, which takes in events from its predecessor pipeline, feeds them into a sub-pipeline for processing, reads back the output from the sub-pipeline, and then pumps it down to the next filter.
The best way to look at sub-pipelines and join filters is to think of them as callees and callers of a subroutine in procedural programming. The input to the joint filter is the subroutine's parameters, the output from the joint filter is its return value.
A root pipeline, such as Port or Timer, cannot be called from join filters.
To get a list of builtin filters and their parameters:
$ pipy --list-filters
$ pipy --help-filters
Another important notion in Pipy is that of contexts. A context is a set of variables attached to a pipeline. Every pipeline gets access to the same set of variables across a Pipy instance. In other words, contexts have the same shape. When you start a Pipy instance, the first thing you do is define the shape of the context by defining variable(s) and their initial values.
Every root pipeline clones the initial context you define at the start. When a sub-pipeline starts, it either shares or clones its parent's context, depending on which joint filter you use. For instance, a link filter shares its parent's context while a demux filter clones it.
To the scripts embedded in a pipeline, these context variables are their global variables, which means that these variables are always accessible to scripts from anywhere if they live in the same script file.
This might seem odd to a seasoned programmer because global variables usually mean they are globally unique. You have only one set of these variables, whereas in Pipy we can have many sets of them (contexts) depending on how many root pipelines are open for incoming network connections and how many sub-pipelines clone their parents' contexts.
Writing a Network ProxySuppose you're running separate instances of different services and you want to add a proxy to forward traffic to the relevant services based on the request URL path. This would give you the benefit of exposing a single URL and scaling your services in the back end without users having to remember a distinct service URL. In normal situations, your services would be running on different nodes and each service could have multiple instances running. In this example, assume you're running the services below, and want to distribute traffic to them based on the URI.
-
service-hi at /hi/* (127.0.0.1:8080, 127.0.0.1:8082)
-
service-echo at /echo (127.0.0.1:8081)
-
service-tell-ip at /ip_/_* (127.0.0.1:8082)
Pipy scripts are written in JavaScript, and you can use any text editor of your choice to edit them. Alternatively, if you have installed Pipy locally, you can use Pipy admin Web UI, which comes with syntax highlighting, auto-completion, hints, as well as the ability to run scripts, all from the same console.
Start a Pipy instance, without any arguments, so the Pipy admin console launches on port 6060. Now open your favorite web browser and navigate to [http://localhost:6060](http://localhost:6060/ to see the built-in Pipy Administration Web UI.
Image by:(Ali Naqvi, CC BY-SA 40)
Create a Pipy programA good design practice is that code and configurations are separated. Pipy supports such modular design through its Plugins, which you can think of as JavaScript modules. That said, you store your configuration data in the config folder, and your coding logic in separate files under the plugins folder. The main proxy server script is stored in the root folder, the main proxy script (proxy.js) will include and combine the functionality defined in separate modules. In the end, your final folder structure is:
├── config│ ├── balancer.json
│ ├── proxy.json
│ └── router.json
├── plugins
│ ├── balancer.js
│ ├── default.js
│ └── router.js
└── proxy.js
1.Click New Codebase, enter /proxy for the Codebase name in the dialog and then click Create.
-
Click the + button to add a new file. Enter /config/proxy.json for its filename and then click Create. This is the configuration file used to configure your proxy.
-
You now see proxy.json listed under the config folder in the left pane. Click on the file to open it and add the configuration shown below and make sure you save your file by clicking the disk icon on the top panel.
{ "listen": 8000, "plugins": [ "plugins/router.js", "plugins/balancer.js", "plugins/default.js" ] }
-
Repeat steps 2 and 3 to create another file, /config/router.json, to store route information. Enter this configuration data:
{ "routes": { "/hi/*": "service-hi", "/echo": "service-echo", "/ip/*": "service-tell-ip" } } -
Repeat steps 2 and 3 to create another file, /config/balancer.json to store your service-to-target map. Enter the following data:
{ "services": { "service-hi" : ["127.0.0.1:8080", "127.0.0.1:8082"], "service-echo" : ["127.0.0.1:8081"], "service-tell-ip" : ["127.0.0.1:8082"] } } -
Now it's time to write your very first Pipy script, which will be used as a default fallback when your server receives a request for which you don't have any target (an endpoint) configured. Create the file /plugins/default.js. The name here is just a convention and Pipy doesn't rely on names, so you can choose any name you like. The script will contain the code shown below, which returns the HTTP Status code 404 with a message of No handler found:
pipy()
.pipeline('request')
.replaceMessage(
new Message({ status: 404 }, 'No handler found'))
7.Create the file /plugins/router.js, which stores your routing logic:
(config =>pipy({
_router: new algo.URLRouter(config.routes), })
.export('router', {
__serviceID: '', })
.pipeline('request')
.handleMessageStart(
msg => (
__serviceID = _router.find(
msg.head.headers.host,
msg.head.path, )
) )
)(JSON.decode(pipy.load('config/router.json')))
-
Create the file /plugins/balancer.js, which stores your load balancing logic as a side-note. Pipy comes with multiple Load Balancing algorithms, but for simplicity, you're using the Round Robin algorithm here.
(config =>
pipy({
_services: (
Object.fromEntries(
Object.entries(config.services).map(
([k, v]) => [
k, new algo.RoundRobinLoadBalancer(v)
]
)
)
),
_balancer: null,
_balancerCache: null,
_target: '',
})
.import({
__turnDown: 'proxy',
__serviceID: 'router',
})
.pipeline('session')
.handleStreamStart(
() => (
_balancerCache = new algo.Cache(
// k is a balancer, v is a target
(k ) => k.select(),
(k,v) => k.deselect(v),
)
)
)
.handleStreamEnd(
() => (
_balancerCache.clear()
)
)
.pipeline('request')
.handleMessageStart(
() => (
_balancer = _services[__serviceID],
_balancer && (_target = _balancerCache.get(_balancer)),
_target && (__turnDown = true)
)
)
.link(
'forward', () => Boolean(_target),
''
)
.pipeline('forward')
.muxHTTP(
'connection',
() => _target
)
.pipeline('connection')
.connect(
() => _target
)
)(JSON.decode(pipy.load('config/balancer.json'))) -
Now write the entry point, or the proxy server script, to use the above plugins. Creating a new code base (step 1) creates a default main.js file as an entry point. You can use that as your main entry point, or if you prefer to go with a different name, feel free to delete main.js and create a new file with the name of your choice. For this example, delete it and create a new file named /proxy.js. Make sure you click the top flag icon to make it the main entry point, to ensure script execution is started when you hit the run button (the arrow icon on the right).
(config =>
pipy()
.export('proxy', {
__turnDown: false,
})
.listen(config.listen)
.use(config.plugins, 'session')
.demuxHTTP('request')
.pipeline('request')
.use(
config.plugins,
'request',
'response',
() => __turnDown
)
)(JSON.decode(pipy.load('config/proxy.json')))
So far, your workspace looks like this:
Image by:(Ali Naqvi, CC BY-SA 40)
To run your script, click the play icon button (4th from right). Pipy runs your proxy script, and you see output similar to this:
Image by:(Ali Naqvi, CC BY-SA 40)
This shows that your proxy server is listening on port 8000 (which you configured in your /config/proxy.json). Use curl to run a test:
HTTP/1.1 404 Not Found
content-length: 10
connection: keep-alive
No handler found
That response makes sense because you haven't configured any target for root. Try one of your configured routes, such as /hi:
$ curl -i [http://localhost:8000/hi](http://localhost:8000/hi)HTTP/1.1 502 Connection Refused
content-length: 0
connection: keep-alive
You get 502 Connection Refused because you have no service running on your configured target port.
You can update /config/balancer.json with details like the host and port of your already running services to make it fit for your use case, or you can just write a script in Pipy to listen on your configured ports, and return simple messages.
Save this code to a file on your local computer named mock-proxy.js, and remember the location where you stored it:
pipy().listen(8080)
.serveHTTP(
new Message('Hi, there!\n')
)
.listen(8081)
.serveHTTP(
msg => new Message(msg.body)
)
.listen(8082)
.serveHTTP(
msg => new Message(
`You are requesting ${msg.head.path} from ${__inbound.remoteAddress}\n`
)
)
Open a new terminal window and run this script with Pipy (change /path/to to the location where you stored this script file):
2022-01-11 18:56:31 [INF] [config]
2022-01-11 18:56:31 [INF] [config] Module /mock-proxy.js
2022-01-11 18:56:31 [INF] [config] ================
2022-01-11 18:56:31 [INF] [config]
2022-01-11 18:56:31 [INF] [config] [Listen on :::8080]
2022-01-11 18:56:31 [INF] [config] ----->|
2022-01-11 18:56:31 [INF] [config] |
2022-01-11 18:56:31 [INF] [config] serveHTTP
2022-01-11 18:56:31 [INF] [config] |
2022-01-11 18:56:31 [INF] [config] <-----|
2022-01-11 18:56:31 [INF] [config]
2022-01-11 18:56:31 [INF] [config] [Listen on :::8081]
2022-01-11 18:56:31 [INF] [config] ----->|
2022-01-11 18:56:31 [INF] [config] |
2022-01-11 18:56:31 [INF] [config] serveHTTP
2022-01-11 18:56:31 [INF] [config] |
2022-01-11 18:56:31 [INF] [config] <-----|
2022-01-11 18:56:31 [INF] [config]
2022-01-11 18:56:31 [INF] [config] [Listen on :::8082]
2022-01-11 18:56:31 [INF] [config] ----->|
2022-01-11 18:56:31 [INF] [config] |
2022-01-11 18:56:31 [INF] [config] serveHTTP
2022-01-11 18:56:31 [INF] [config] |
2022-01-11 18:56:31 [INF] [config] <-----|
2022-01-11 18:56:31 [INF] [config]
2022-01-11 18:56:31 [INF] [listener] Listening on port 8080 at ::
2022-01-11 18:56:31 [INF] [listener] Listening on port 8081 at ::
2022-01-11 18:56:31 [INF] [listener] Listening on port 8082 at ::
You now have your mock services listening on ports 8080, 8081, and 8082. Do a test again on your proxy server to see the correct response returned from your mock service.
SummaryYou've used a number of Pipy features, including variable declaration, importing and exporting variables, plugins, Pipelines, sub-pipelines, filter chaining, Pipy filters like handleMessageStart, handleStreamStart, and link, and Pipy classes like JSON, algo.URLRouter, algo.RoundRobinLoadBalancer, algo.Cache, and others. For more information, read the excellent Pipy documentation, and through Pipy's admin web UI, and follow the step-by-step tutorials that come with it.
ConclusionPipy from Flomesh is an open source, extremely fast, and lightweight network traffic processor. You can use it in a variety of use cases ranging from edge routers, load balancing and proxying (forward and reverse), API gateways, static HTTP servers, service mesh sidecars, and many other applications. Pipy is in active development and is maintained by full-time committers and contributors.
Pipy is an open source, extremely fast, and lightweight network traffic processor. It has a variety of use cases including edge routers, load balancing and proxying, API gateways, static HTTP servers, service mesh sidecars, and many other applications.
Image by:Image by Mapbox Uncharted ERG, CC-BY 3.0 US
Containers Networking What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Travel off the grid and still send emails with putmail
Automation is a hot topic right now. In my day job as an SRE part of my remit is to automate as many repeating tasks as possible. But how many of us do that in our daily, not-work, lives? This year, I am focused on automating away the toil so that we can focus on the things that are important.
In "normal times" my wife and I travel a lot. And as anyone who similarly travels a lot knows, connectivity can be very expensive. We were recently on a cruise and the "Premium" ship-board internet cost upwards of $200 for the full 7 days, and even then it had all the drawbacks of satellite internet in terms of speed and congestion. So to make my life easier, I set up offlineimap to download my mail, dovecot to let me use an IMAP client to read my mail, and msmtp to send mail from my various accounts. I covered much of this setup in 2020, and the rest is documented in many places online.
More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resourcesWhat is not often discussed is that you still need to be online to send mail. The most common recommended solution for sending mail is msmtp, and it always rejects mail if it can't connect to the desired host. But it does have a daemon option (msmtpd) that can be used to accept local SMTP connections and forward them to another program. By default, this is msmtp itself. But again, I don't want to send things now, I want to send things when I have a connection, which is when I stumbled upon putmail.
Putmail is a set of Python scripts that sends mail to pre-configured smtp servers based on the From address in a mail message. It is stable (unmodified since 2011), and just works. If you, like me, have multiple email addresses which have to be sent via multiple SMTP relays, this is just the thing you need. Each email address you send from has its own configuration file, and since putmail decides on which to use based on the message itself, there's no need to have to set up multiple sending setups in a single mail client.
As an example, to send with a gmail account, you would create the file .putmail/yourname@gmail.com and fill in the following information.
yourname@gmail.com putmail configuration[config]
email = yourname@gmail.com
server = smtp.gmail.com
port = 587
username = yourname@gmail.com
password = XXXXXXXXXXXXXXXXX
tls = on
And that's it. Configure a mail client to send via putmail.py instead of sendmail or msmtp and send a message.
The next best things are the putmail_enque.py and putmail_dequeue.py scripts. The first takes an email and stores it to send later. The second loops through the queue and delivers the mail. By specifying putmail_enqueue.py as the program msmtpd runs, I can "send" an email now, and it just waits for me to run putmail_dequeue.py later. Here is my msmtpd startup command, specifying putmail_enqueue.py as the item to be used for mail delivery.
msmtpd --port=1025 --log=/tmp/msmtpd.log --command='putmail_enqueue.py -f %F' -I use the following script as a presynchook in offlineimap to check to see if I am connected, and if so, send mail.
#!/bin/bashecho Sending queued messages \(if any\)
QUEUEDMAIL=$(find $HOME/.putmail/queue -type f | wc -l)
if [ $QUEUEDMAIL -ne 0 ]; then
ping -n -c 1 imap.gmail.com >/dev/null 2>/dev/null
if [ $? -eq 0 ]; then
putmail_dequeue.py
fi
fi
After all that, I can use any mail client I wish and send mail with a standard SMTP call to localhost:1025 and have it delivered next time I'm connected to the internet. And the best part is, I don't have to change my workflow for email if I'm home or traveling — it all just happens automatically in the background.
Configure your mail client to automatically send emails next time you have an internet connection.
Email What to read next Manage your Gmail filters from the Linux command line This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.4 ways you can preserve the Earth's livability with open source
Earth's uniqueness in creating a stable environment for life in a completely hostile space is a miracle. Various life forms have taken billions of years to build up the natural resources humans depend on, such as a protective atmosphere, fertile soil, stable weather, and clean drinking water.
As a movement to democratize technology development and knowledge creation, open source has the potential to become the central driver in preserving this stability. Here are four different ways you can preserve the livability of the planet by using, supporting, and participating in open source projects, and more than 20 resources to help you get started.
Measure, optimize, and reduceBefore taking action, it is important to assess your impact on the environment in a scientific way. With more accurate information, you can focus your efforts where the most potential lies. Open science discloses the models behind these measurements, which reduces uncertainties in the long term through continuous improvement and observation. Open source tools make it possible to remain objective and make meaningful comparisons to guide decision making.
These decisions happen constantly, both in the consumption of everyday resources and in the use and development of digital products. Software developers find their work in digital products all over the world. Measuring and reducing the energy consumption of these products is the goal of open source tools like Scaphandre, Cloud Carbon Footprint, and kube-green. These tools help you track power consumption, estimate carbon emissions, and even shut down resources when they are not needed.
Image by:Cloud Carbon Footprint is an open-source project, sponsored by Thoughtworks Inc
(Cloud Carbon Footprint, Apache License, Version 2.0)
The open source platform electricityMap calculates and predicts the carbon intensity of electric power consumption within electrical grids worldwide on an hourly basis. You or your business can take action by using this information to shift computation to times when there is a high share of renewable energy sources in the electricity grid, thereby reducing emissions.
The open source community of OpenEnergyMonitor enables people to monitor and understand the electrification of their homes. Open hardware and open source tools created by OpenEnergyMonitor track and log usage data from multiple categories, including electric vehicles, heat pumps, and photovoltaics. Their learning platform offers a deep understanding of the possibilities for increasing the use of sustainable energy in your household and becoming energy independent. At the heart of it all is the emonPi energy monitor and base station, which can bring together information about your energy consumption from multiple sensor nodes.
Image by:The OpenEnergyMonitor Solar PV monitor provides real-time and historical information on solar generation and on-site consumption.
(OpenEnergyMonitor, CC BY-SA 4.0)
The Earth is a highly complex system, which makes it vitally important to better understand the impact humans have on this system. Several open science and open source projects are working to improve transparency and cooperation in this area. Various tutorials offer beginners an introduction to the programming languages and tools used in this field. These tutorials allow you to improve your programming skills while simultaneously learning more about the environment.
An Introduction to Earth and Environmental Data Science provides a beginner-friendly introduction to the use of open source Python tools, how these tools can be used, and data science in general. Instead of introducing the basics of Python with routine robotics projects, learners discover how to use Python and programming tools to interpret data about the planet.
Image by:Understanding and representing earth observation data, such as this map of sea ice concentration, is a central part of the Introduction to Earth and Environmental Data Science course.
(Tobias Auspurger, CC BY-SA 4.0)
The Climate Laboratory provides the perfect starting point for getting a deeper understanding of climate change and the physical science behind it. Like the previous example, this course includes tutorials to use Python, one of the main tools for climate scientists today. The textbook is a living document that has been created around climlab to provide a powerful but entry-level friendly tool for climate modeling.
Image by:You can learn to make maps illustrating surface air temperature anomalies due to CO2 doubling.
(Tobias Auspurger, CC BY-SA 4.0)
Earth Lab provides Earth data science lessons that will help you learn how to work with data in the R and Python programming languages. All course material is open source. Nonspecialists who enroll in the Earth data analytics online course help ensure that understanding how the planet works is not just an academic exercise but also a field of knowledge used in all parts of society.
Observe the planetSatellites provide a unique tool to map the state of the planet. Even small changes in our ecosystems such as greenhouse gas emissions, land use change, biodiversity loss, soil drought, or water pollution can be measured with high temporal and spatial resolution.. The data collected here are calibrated to the highest scientific standards and therefore provide an objective view of Earth. Recent years have seen a massive democratization of data and software tools in this field. NASA and ESA have published many data sets and tools from various satellite missions under a free license.
The condition of even the most remote forests can be assessed with the help of satellite images in different spectra. All kinds of forest health indicators can be derived from data, like deforestation risk, wildfire risk, or tree detection. By using and improving these tools, you can actively contribute to preserving this ecosystem. Only by measuring and disclosing such information can environmentalists demonstrate that habitats are being lost.
You can also learn how to process satellite data using modern image processing algorithms such as neural networks. One of the most robust communities in this field is Global Forest Watch. Here, multiple datasets are combined to give an interactive overview of the health of forests worldwide.
Image by:Spatial probability of deforestation on Guadeloupe
(Tobias Auspurger, CC BY-SA 4.0)
Projects like Resource Watch and Radiant Earth provide entry-level platforms for people anywhere to understand natural resources based on Earth-observation technology. NASA provides its own open source platform, NASA Harvest, to map the state of crops, providing essential information about food security. Another way to get insights about possible hunger crises is the measurement of soil moisture, which is also possible from space with packages like smapr.
The ever-increasing resolution of satellite data in recent years means that even the impact of individual companies on the environment can be measured. This new field of science is called spatial finance, and its purpose is to guide business development and investment in alignment with good sustainability practices. The first open datasets in this field track the locations and companies involved in cement production, iron and steel production, and global coal, oil, and gas supply chains. The Carbon Mapper platform shows how to build pressure on companies by disclosing methane and carbon dioxide sources.
Image by:Global coal, oil, and gas asset-level data
(Lucas Kruitwagen, MIT License)
Environmental catastrophes are often accompanied by a cover-up and manipulation of scientific findings and information. This applies to climate change as well as local environmental impacts. UN Secretary-General Antonio Guterres has given a clear warning following the release of the international climate change report, which shows no trend reversal in global warming and the loss of biodiversity in 2022.
We are on a pathway to global warming of more than double the 1.5°C limit agreed in Paris. Some government and business leaders are saying one thing, but doing another. Simply put, they are lying. And the results will be catastrophic. This is a climate emergency.
Image by:With the help of pyam you can process the data behind the latest UN report and thus better understand the different scenarios.
(pyam, Apache license)
One of the easiest ways to change course is to consistently demand open science concerning environmental issues. Openness is a key indicator of sustainable development, because anyone pursuing truly sustainable intentions is interested in refining their conclusions through open scientific discourse. Openness will also help prevent greenwashing, or the practice of promoting a company's sustainability efforts through marketing while scientific findings and conclusions about environmental impacts are downplayed or held private.
The next time you buy products that claim to be carbon neutral, ask for independent studies and data. If a politician demands questionable measures for environmental protection, ask for open model calculations and open access studies on this topic. Do you want to invest in a sustainable investment fund? Ask for open scientific studies, data, and models on why this investment is sustainable. Demanding openness in all these ways helps limit greenwashing and ensures that companies' environmental impacts are verifiable, accountable, and traceable.
Why actions rooted in open source culture can make a big impact.
Image by:Seth Kenlon, CC BY-SA 4.0
Science Data Science What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.5 benefits of switching from Google Analytics to Plausible
Google Analytics (GA) has been the industry standard web analytics tool for about as long as there have been analytics tools. Nearly every brief that my WordPress agency receives specifies that GA must be installed. And there is rarely any debate around whether it's the best tool for the job.
My team at Wholegrain Digital has had concerns about GA in terms of privacy, General Data Protection Regulation (GDPR) compliance, performance, user experience, not to mention Google as a global advertising, and search monopoly. However, we continued using GA for 99% of our projects because we didn't feel that there was a strong enough alternative.
Well, that has changed. We've made a decision that GA will no longer be our default analytics tool. Instead, our default analytics tool will be Plausible.
In this article, I'll outline why we consider Plausible a better choice as a default analytics solution, the compromises to be aware of, and how it will impact our clients.
Why use Plausible instead of Google AnalyticsWe believe that using Plausible over GA is not a purely ideological choice. There are in fact a number of tangible benefits that make it an objectively better product in many cases. Let's take a look at some of these benefits.
More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources PrivacyOne of Plausible's headline benefits is that it has been designed as a privacy-first analytics tool. This might sound like an ideological factor, but it has real practical implications too.
Plausible only collects anonymous user data, and does not use cookies. This means that unlike most analytics solutions, it complies with both the GDPR and the European Cookie Law, so it's a tick in the box for legal compliance. It's also hosted in the EU and the data is owned by you so it doesn't get shared with any other organizations, which is another tick.
This has a positive knock-on effect for user experience, because without privacy-invading cookies, you don't need to implement a cookie banner on the website (unless you use cookies for other things). Considering that the first few seconds of a website visit are the most critical to brand experience and conversion rates, cookie banners are arguably the single most damaging thing to the success of any online service. Avoiding these banners puts your brand and message first, helps you make a good first impression, and increases conversion rates.
Simpler user interfaceThe GA dashboard was never a great case study in intuitive design, but as the functionality of GA has expanded, so too has the complexity of the interface. I'm increasingly hearing frustrations from even experienced web marketers that they sometimes struggle to find basic data inside GA.
Plausible has taken the opposite approach and focused on ease of use over quantity of features. The dashboard is beautifully designed to showcase the data you want in a way that is easy to find and understand. It's what most people wish GA to be and it's a breath of fresh air.
Image by:(Tom Greenwood, CC BY-SA 4.0)
This is not just a pleasantry. The reality is that if a tool is easy to use then you will use it, and if it is hard to use then you won't. The simplicity of the Plausible interface means that even though it has less features, many users are likely to get a lot more insights and value from their analytics.
Better web performanceThe Plausible tracking script is the smallest we are aware of for any analytics service, coming in at less than 1kb compared to 17kb for GA and even more if you're using Google Tag Manager.
Better web performance improves user experience, helps your website to rank better in search engines and improves conversion rates. Web performance matters, and when it comes to web performance, every kilobyte matters.
Reduced environmental impactPlausible's tiny tracking script isn't just good for web performance, it's also good for the environment. At Wholegrain we are world leaders in sustainable web design and understand that every bit of wasted data is also wasted energy and carbon emissions.
This adds up. As a minimum, switching to Plausible would save 16kb per visitor, so for a website with a modest 10K visitors per month this would be 160MB of data per month, or 2GB per year. A back of an envelope calculation using the latest methodology that we have developed for website carbon calculations put this at about the equivalent of 800 grams of CO2. Now multiply that up by the millions of websites running worldwide and you are talking about significant amounts of wasted energy and unnecessary carbon emissions.
[ Read also: 5 open source tips to reduce waste in web design ]
But it doesn't stop there. Analytics tools consume energy on the end user's device as they harvest data, in the transmission networks as they send that data back to the data center, and in the data center to store it all for eternity. By tracking less, Plausible is using less energy in its operation than bloated analytics tools such as GA that are tracking many more metrics that most people don't need.
Plausible also hosts the data in data centers with a commitment to using renewable sources of electricity, which is great (though to be fair Google also does that), and they donate 5% of their revenue to good causes, so the money you are paying is also helping support environmental and social projects.
More accurate dataFinally, Plausible breaks one of the greatest myths about GA, which is that it is accurate. Plausible's own research and our own experiences with client websites show that GA can significantly under report data, sometimes by as much as 50-60%.
When you look at GA and you think you are looking at all of your website data, you are most likely only looking at a portion of it. This is because GA is more likely to be blocked by ad blockers and privacy friendly web browsers, but perhaps more significantly because GA should not be tracking your visitors unless they have accepted cookies, which many visitors do not.
Plausible therefore offers us a simple way to get a more complete and accurate view of website visitors. Needless to say, if you are a digital marketing manager and you want to make yourself look good, you want to use the analytics service that reports all of your visitors and not just some of them.
Are there any downsides?There are very few downsides to using Plausible. If you are a GA power user and genuinely need a lot of its more advanced functionality then Plausible is not going to meet all of your needs. Plausible can track events, conversions, and campaigns but if you need more than that then you'll need another tool.
The only other notable downside to Plausible is that it is not free. There is a monthly fee but it is very reasonable, starting at $6 per month. It's a small price to pay for such a good tool, to protect the privacy of your website visitors and to maintain ownership of your data.
What are the impacts?Here at Wholegrain Digital we are implementing Plausible in a way that we hope will offer many of our clients the benefits listed above without the downsides. Nobody will be forced to stop using GA, but we will be talking about the benefits of Plausible proactively with our clients and including it in our proposals as the default option.
In many cases, Plausible is not just capable of meeting a project's requirements, but is objectively a better solution than GA, so we'll be trying to help our clients to reap the benefits of this.
In cases where advanced GA features are required, we will of course stick with GA for now. And in some cases, we may run GA and Plausible in parallel. This has a small overhead in terms of carbon footprint and performance, so it isn't our first choice. However, in some cases it may provide a necessary bridge to help our clients gain confidence in Plausible before making the leap, or help them gain a complete data set while ensuring compliance with privacy laws.
As for the costs, Plausible is a very good value for what it offers, but we will be offering it free of charge to our clients for any websites cared for on our WordPress maintenance service. We hope that this will help lower the barrier to entry and help more of our clients and their website visitors gain the benefits of Plausible.
It is Plausible!I hope I've convinced you that Plausible is not just a viable but a really great alternative to GA. If you need any advice on how to figure out the best analytics solution for your website or how to ensure that you are GDPR compliant without compromising user experience, get in touch with me.
This article originally appeared on the Wholegrain Digital blog and has been republished with permission.
Plausible is an open source alternative to Google Analytics.
Image by:Opensource.com
Alternatives What to read next Why choose Plausible for an open source alternative to Google Analytics This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Use Composer to require Git repositories within PHP projects
The dependency management tool Composer provides multiple ways to include Git repositories within a PHP: Hypertext Preprocessor (PHP) project.
In many cases, repositories have been created on Packagist, so requiring them with Composer is very straightforward. But what do you do when a repository has not been created as a package on Packagist? You use Composer to require the package directly from the repository. This article explains how.
Note: Some of the terminology in this post is confusing because multiple words are used to describe different things. Here is a quick vocabulary list that will help:
- Project: The custom software you are building. This can be a website, a command-line utility, an application, or anything else you dream up.
- Package: Any third-party software you want to download and use within your project. It can be a library, Drupal theme, WordPress plugin, or any other number of things.
- Git repository: Also called the Git repo, this is the version-control host for a package. Common hosts include GitHub, GitLab, or Bitbucket, but any URL-accessible Git repository will work for this tutorial.
- Composer repositories: In a composer.json file, there is an optional property named "repositories." This property is where you can define new places for Composer to look when downloading packages.
When adding a Git repo to your project with Composer, you can find yourself in two situations: Either the repo contains a composer.json file, which defines how the repo should be handled when required, or it does not. You can add the Git repository to your project in both cases, with different methods.
Git repo with composer.jsonWhen a repository includes a composer.json file, it defines aspects of itself that are important to how Composer manages the package. Here is an example of a simple composer.json file a package may include:
{ "name": "mynamespace/my-custom-library", "type": "library" }This example shows two important properties that a composer.json file can define:
- Name: The package's namespaced name. In this case, "mynamespace" is the namespace for the package "my-custom-library."
- Type: The type of package the repo represents. Package types are used for installation logic. Out of the box, Composer allows for the following package types: library, project, metapackage, and composer-plugin.
You can verify this by looking at the composer.json file of any popular GitHub project. They each define their package name and the package type near the top of the file. When a repository has this information defined in its composer.json file, requiring the repository within your project is quite simple.
More on Git What is Git? Git cheat sheet Markdown cheat sheet New Git articles Require a Git repository that has a composer.json fileAfter you've identified a Git repository with a composer.json file, you can require that repository as a package within your project.
Within your project's composer.json file, you need to define a new property (assuming it doesn't exist already) named "repositories." The value of the repositories property is an array of objects, each containing information about the repository you want to include in your project. Consider this composer.json file for your custom project.
{ "name": "mynamespace/my-project-that-uses-composer", "repositories": [ { "type": "vcs", "url": "https://github.com/mynamespace/my-custom-library.git" } ], "require": { "mynamespace/my-custom-library": "dev-master" } }You are doing two important things here. First, you're defining a new repository that Composer can reference when requiring packages. And second, you're requiring the package from the newly defined repository.
Note: The package version is a part of the require statement and not a part of the repository property. You can require a specific branch of a repo by choosing a version named "dev-".
If you were to run composer install in the context of this file, Composer would look for a project at the defined URL. If that URL represents a Git repo that contains a composer.json file that defines its name and type, Composer will download that package to your project and place it in the appropriate location.
Custom package typesThe example package shown above is of the type "library," but with WordPress and Drupal you're dealing with plugins, modules, and themes. When requiring special package types in your project, it's important to install them in specific locations within the project file structure. Wouldn't it be nice if you could convince Composer to treat these types of packages as special cases? Well, you're in luck. There is an official Composer plugin that will do that for you.
The Installers plugin for Composer contains the custom logic required for handling many different package types for a large variety of projects. It is extremely helpful when working with projects that have well-known and supported package installation steps.
This project allows you to define package types like drupal-theme, drupal-module, wordpress-plugin, wordpress-theme, and many more, for a variety of projects. In the case of a drupal-theme package, the Installers plugin will place the required repo within the /themes/contrib folder of your Drupal installation.
Here is an example of a composer.json file that might live within a Drupal theme project as its own Git repository:
{ "name": "mynamespace/whatever-i-call-my-theme", "type": "drupal-theme", "description": "Drupal 8 theme", "license": "GPL-2.0+" }Note that the only meaningful difference here is that the type is now drupal-theme. With the drupal-theme type defined, any project that uses the Installers plugin can easily require your repo in its Drupal project, and it will be treated as a contributed theme.
Require any Git repository with ComposerWhat happens when the repo you want to include in your project does not define anything about itself with a composer.json file? When a repo does not define its name or type, you have to define that information for the repo within your project's composer.json file. Take a look at this example:
{ "name": "mynamespace/my-project-that-uses-composer", "repositories": [ { "type": "package", "package": { "name": "mynamespace/my-custom-theme", "version": "1.2.3", "type": "drupal-theme", "source": { "url": "https://github.com/mynamespace/my-custom-theme.git", "type": "git", "reference": "master" } } } ], "require": { "mynamespace/my-custom-theme": "^1", "composer/installers": "^1" } }Notice that your repository type is now "package." That is where you will define everything about the package you want to require.
Create a new object named "package" where you define all the essential information that Composer needs to know to be able to include this arbitrary git repo within your project, including:
- Name: The namespaced package name. It should probably match the repository you're requiring but doesn't have to.
- Type: The type of package. This reflects how you want Composer to treat this repository.
- Version: A version number for the repo. You will need to make this up.
- Source: An object that contains the following repository information:
- URL: The Git or other version-control system (VCS) URL where the package repo can be found
- Type: The VCS type for the package, such as git, svn, cvs, and so on
- Reference: The branch or tag you want to download
I recommend reviewing the official documentation on Composer package repositories. Note that it is possible to include zip files as Composer packages as well. Essentially, you are now responsible for all parts of how Composer treats this repository. Since the repository itself is not providing Composer with any information, you are responsible for determining almost everything, including the current version number for the package.
This approach allows you to include almost anything as a Composer package in your project, but it has some notable drawbacks:
- Composer will not update the package unless you change the version field.
- Composer will not update the commit references. If you use master as a reference, you will have to delete the package to force an update, and you will have to deal with an unstable lock file.
Maintaining the package version in your composer.json file isn't always necessary. Composer is smart enough to look for GitHub releases and use them as the package versions. But eventually you will likely want to include a simple project that has only a few branches and no official releases.
When a repository does not have releases, you will be responsible for deciding what version the repository branch represents to your project. In other words, if you want composer to update the package, you will need to increment the "version” defined in your project's composer.json file before running composer update.
Overriding a Git repository's composer.jsonWhen defining a new Composer repository of the type package, you can override a package's own composer.json definitions. Consider a Git repository that defines itself as a library in its composer.json, but you know that the code is actually a drupal-theme. You can use the above approach to include the Git repository within your project as a drupal-theme, allowing Composer to treat the code appropriately when required.
Example: Require Guzzle as a drupal-theme just to prove that you can.
{ "name": "mynamespace/my-project-that-uses-composer", "repositories": [ { "type": "package", "package": { "name": "mynamespace/guzzle-theme", "version": "1.2.3", "type": "drupal-theme", "source": { "url": "https://github.com/guzzle/guzzle.git", "type": "git", "reference": "master" } } } ], "require": { "mynamespace/guzzle-theme": "^1", "composer/installers": "^1" } }This works! You've downloaded the Guzzle library and placed it within the /themes folder of your Drupal project. This is not a very practical example, but it highlights how much control the package type approach provides.
SummaryComposer offers plenty of options for including arbitrary packages within a project. Determining how those packages are included in the project primarily comes down to who defines the package information. If the Git repository includes a composer.json file that defines its name and type, you can have Composer rely on the repository itself for the definition.
But if you want to include a repository that does not define its name and type, then it is up to your project to define and maintain that information for your own internal use. Alternatively, if a repository doesn't define a composer.json file, consider submitting a pull request that adds it.
This article originally appeared on the Daggerhart Lab blog and is republished with permission.
This dependency management tool makes it easier to require a repository even when it hasn't been created as a package.
Image by:CC BY 3.0 US Mapbox Uncharted ERG
Git What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.5 reasons to apply for B Corp certification
Wholegrain Digital just passed its fifth anniversary of being a certified B Corp. B Corp is one of the most robust independent certifications assessing whether a business is run responsibly in terms of its environmental impact, its treatment of workers and suppliers, its impact on communities, and its governance.
As one of the longest-standing B Corps in the UK, we're increasingly asked for advice by companies considering going through the B Corp assessment process, known as the B Impact Assessment. There is one question that seems to come up more than any other, and it is, "Is there a business case for becoming a B Corp?"
In this article, I'll share my own perspective on that question, based upon what I've experienced in our business and what I've observed from other B corps that I know.
Do it for the right reasonsFirst of all, I need to highlight what B Corp is not. It's not a marketing gimmick. There are loads of certifications and badges that companies can get to make themselves look good, but which don't really mean anything.
B Corp is different in that it's both independent and really hard to obtain. The difficulty of the assessment is exactly what makes it meaningful. Therefore, if you're thinking that the business benefit is that it's a cheap marketing gimmick, then you're headed in the wrong direction. B Corp certification can have significant benefits to any business, but only if it's pursued for the right reasons and with a genuine commitment to become a more responsible business.
So, let's look at the real benefits.
More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources B Corp helps small businesses grow upFrom day one, we always tried to run Wholegrain Digital as a responsible business but it was hard to ever know how well we were doing. When we first started the B Impact Assessment after being in business for about 9 years, we quickly realized that we needed to get more organized about how we manage things across the business.
We needed to put some structures and policies in place and properly document evidence of what we're doing. As a small creative business, this was something that we were not very good at and so the process of going through the B Impact Assessment helped us to grow up as a business. It not only gave us better visibility of our performance, but it also encouraged us to put in place the structures that the business needed in order to grow and progress.
This benefit is probably mostly applicable to small businesses, but it's one that can help take a business over that hurdle from a small scrappy startup to a serious, mature business.
B Corp guides you to better performanceThe B Impact Assessment is largely multiple choice, with evidence being needed to prove that you have delivered the options that you select. The beauty of this multiple choice format is that for every issue, you can see a variety of options and you can see which ones are considered more responsible than others.
This is a great learning tool because in most cases, you probably won't be able to select the "best" option, but will then know what the best option is considered to be. With this knowledge, you can take action to improve in the future. The assessment process therefore helps accelerate progress and give you some targets to aim for.
B Corp helps create a culture of positive changeAs mentioned, the impact assessment process can help you see where you could create more positive impact, but the positive change that can come from certifying as a B Corp can go well beyond the things actually included in the assessment. The process gets you thinking more deeply about where you have impact. It also sends a signal to everyone in the organization that there is a deep commitment to creating positive impact.
The result is a stronger culture of positive change from which new ideas and initiatives emerge. For example, we've implemented schemes such as Climate Perks, extra holiday for using renewable energy at home and a vegetarian food policy, even though they don't gain us any points in the B Impact Assessment. More significantly, the thought process that was sparked by our first attempt at the B Impact Assessment led us to research the environmental impact of digital technology and go on to become world leaders in the emerging field of sustainable web design.
The innovation inspired by this pursuit has led us to learn new skills and constantly improve the quality of the work that we're delivering, not just environmentally, but across the board. It's inspired, motivated and educated us to perform at the highest level, making us truly proud of the work that we're doing and in turn, helping us to be more commercially successful too.
[ Download the eBook: The Open Organization Guide to IT Culture Change ]
B Corp shows that you are seriousTwo of the benefits that I often hear mentioned are that B Corp certification can help attract talented staff and new clients. In my experience running a digital agency, I would say that this is partially true. I don't think it's a magic bullet in delivering either of these things, but it certainly can help.
Awareness of B Corp certification among the general public is still relatively low in the UK so many people won't think much of it. However, for anyone who cares about creating a positive impact, it's a great way of demonstrating that you are serious about it. It helps you to earn their trust and respect and this can definitely help in attracting employees and clients with similar values.
B Corp is a real communityThere are many things out there today that claim to be communities, but almost none of them are. However, when we first achieved B Corp certification, we were amazed to find ourselves welcomed into a community of business leaders who genuinely cared about creating positive impact and wanted to support other businesses to do the same.
The openness and enthusiasm to share and collaborate was refreshing. We've made some great friends, had some amazing shared experiences, and learnt a huge amount from others. This has helped us to stay engaged and motivated in the journey of evolving our own business, as well as providing practical support where we needed it. We are a more successful business as a result.
When we joined the B Corp community, it was very small in the UK, so it was a very tight-knit group. The pandemic paused most in-person activities and some of this community energy was lost temporarily. I hope it will soon return to full force.
As the UK B Corp movement is now reaching momentum, with 700 companies certified to date and a huge waiting list, there is of course a risk that the true sense of community gets diluted. I therefore hope that the sense of community will be able to be maintained over the long term.
[ Read also: How to make community recognition more inclusive ]
So is being a B Corp good for open source business?As Managing Director, I have found that becoming a B Corp has been hugely beneficial to us as at Wholegrain Digital and has contributed to our success in recent years.
I believe that it should only be pursued if you truly want to run a more responsible business. It's a lot of work that won't immediately pay for itself but if you're in it for the long term, it can pay dividends for people, the planet, and be financially profitable too.
This article originally appeared on the Wholegrain Digital blog and has been republished with permission.
B Corp certification shows that your open source business runs responsibly in terms of its environmental impact, its treatment of workers and suppliers, its impact on communities, and its governance.
Image by:Opensource.com
Business What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.When open source meets academic publishing: Platinum open access journals
Academics routinely give away their work to companies for free—and then they buy it back! Can you imagine a farmer giving away free food and then paying to get it back for dinner? Probably not. Yet academics like me have been trapped for decades in a scheme where we give free work in exchange for job security and then pay millions of dollars a year to read our own writing.
Fortunately, this is changing. The results from a study I just finished show that it is possible for academics to get job security without paying for it. My study found hundreds of journals that are platinum open access (OA)—that is, they require neither the author nor the readers to pay for peer-reviewed work—yet still carry the prestige and readership to help academics succeed in their careers.
This trend is exploding: The Directory of Open Access Journals lists over 17,300 journals that offer a means of OA at some level, and over 12,250 have no article-processing charges (APCs). I used a handy open source Python script to compare this list to a list of journals ranked by the frequency with which their published papers are cited in other articles (The Journal Impact Factor List). It is clear that the last few years have seen a growing trend towards both OA in general and platinum OA specifically. These trends have the potential to accelerate science while helping prevent academic servitude.
The academic's dilemmaAcademics are generally pretty intelligent, so why have they engaged in this disadvantageous system for so long? Simply put, academics have been caught in a trap: In order to keep their jobs and get tenure, they need to publish in journals with a high impact factor. An impact factor is a metric based on the mean number of citations to articles published in the last two years in a given journal, as indexed by the proprietary Web of Science. Impact factors are a prestige metric for academics.
Historically, academic publishing has been dominated by a handful of major publishers that used subscription-based business models. In this model, academic authors write articles, peer-review articles, and often do the editing of these articles—all for free. The articles are published under copyright owned by the major publishing companies. Then either the same academics pay to read these articles on an individual basis (~US $35/article), or their university libraries pay to subscribe to all of the articles in a journal. These costs can be astronomical: often over US $1 million per year for all titles from a single publisher.
More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resourcesThis system is senseless for many obvious reasons. Scientific progress is bogged down by restricting access to copyrighted scientific literature squirreled away behind paywalls. It is hard to do state-of-the-art research if you do not know what it is because you cannot read it. Scientists are divided into those who can afford access to the literature and those who cannot. Academics in the developing world often struggle to pay, but even well-endowed Harvard University has taken action to rein in its yearly journal expenses.
Costs to authors are similarly high. APC values range from a few hundred dollars to jaw-dropping thousands of dollars per article. APCs can be particularly damaging for some disciplines that are less well funded, such as the humanities and social sciences (as compared to physical and medical sciences or engineering). Substantial APCs also reinforce the wealth gap in academia, making professional success dependent on having income to invest in publishing. Is there another profession that asks workers to pay money to make products for others?
Open access to the rescue!This problem can be solved by the OA movement, which advocates for making all academic literature freely accessible to everyone. There is an unmistakable rise in OA publishing: It now makes up nearly a third of the peer-reviewed literature.
The benefits of OA are twofold. First, OA is a benefit to science overall, because it provides a frictionless means of reading the state of the art for making significant advancements in knowledge. Second, from an individual academic's point of view, OA provides the pragmatic advantage of enabling the broadest possible audience of their writing by making it freely and easily available on the internet.
Funders have begun to demand OA for these reasons, particularly public funders of science. It is hard to argue that if the public funds research, they should have to pay a second time to read it.
Where is academic publishing now, and where it is going?Conventional publishers still have control of this situation, largely because of the perception that they have a monopoly on journals with an impact factor. Despite the disadvantages of publishing the traditional way, many academics continue to publish in subscription-based journals or pay high APCs, knowing that publication in high impact factor journals is vital for demonstrating expertise for grants, tenure, and promotion.
A few years ago, academics simply had no choice: They could either publish in a journal with an impact factor or publish OA. Now they can publish OA and still get the benefits of an impact factor in one of three ways:
- Green OA: Publish in a traditional way and then self-archive by uploading preprints or accepted versions of papers into an open repository or server. Some schools have an institutional repository for this purpose. For example, Western University has Scholarship@Western, where any of their professors can share their work. Academics without their own institutional repos can use servers like preprints.org, arXiv, or OSF preprints. I also use social media for academics, like Academia or ResearchGate, for self-archiving. This can be complex to navigate because publishers have different rules, and it is somewhat time consuming.
- Gold OA: Publish in a growing list of journals with impact factors that make your paper freely available after publication but require an APC. This method is easy to navigate: Academics publish as usual and OA is built into the publishing process. The drawback is that funds going to APCs may be diverted from research activities.
- Platinum OA: Publish in platinum OA journals with an impact factor. No one pays either to read or to publish. The challenge here is finding a journal in your discipline that fits this criterion, but that continues to change.
There are tens of thousands of journals, but only a few hundred platinum OA journals with impact factors. This may make it hard for academics to find a good fit between what they study and a journal that matches their interests. See the Appendix in my study for the list, or use the Python script mentioned above to run updated numbers for yourself. The number of platinum OA journals is growing quickly, so if you do not find something now you may have some solid journals to choose from soon. Happy publishing!
Academics can now publish free, read free, and still stay on track for professional success.
Image by:Opensource.com
Education What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.