Drop swap for zram on Linux
I spend a lot of time playing (I mean working) on my computers, and I've found a lot of interesting things. One that has most recently come to my attention is the zram0 device. I first noticed it when working on one of my Opensource.com articles several months ago. It showed up in the output from the lsblk command:
# lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 931.5G 0 disk
├─sda1 8:1 0 600M 0 part
[...]
zram0 252:0 0 8G 0 disk [SWAP]
It's identified as swap space, which is what first piqued my curiosity, so I did some exploration. Zram was originally called "compcache," which stands for "compressed cache." It turns out that zram is a tool for creating an in-RAM compressed cache, specifically for use as swap space.
But Why?
More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articlesWhen I began researching zram, all I found were a couple of basic articles about using zram for swap space. At first, this seemed a bit counterintuitive to me. After all, if you're running out of RAM and you swap pages into a virtual drive in RAM, what's gained?
I then found the Fedora Project wiki page that proposed the use of Swap on zram. The proposal says: "Swap is useful, except when it's slow. zram is a RAM drive that uses compression. Create a swap-on-zram during start-up. And no longer use swap partitions by default."
The rest of the page is about details, benefits, side effects, and feedback.
Zram for swap space on LinuxUsing zram for swap space is intended to do the same thing as regular partition-based or file-based swap space. When memory pressure becomes too great, some of the least recently used data is moved to swap space. On average, it's compressed to about 50% of its original size, and placed in zram space in RAM. This is much faster than storing those memory pages on a hard drive and frees up the RAM it was using for other use.
Saving on swapI tried to find revised recommendations for how much swap or zram swap to configure. This led me back to a reassessment of swap, and my previous article, What's the right amount of swap space for a modern Linux system? As far as I can tell from the most current documentation for RHEL and Fedora, the recommended amount of swap space has not changed. That documentation, however, ignores the use of zram.
However, the tables in that previous article still provide a good starting point for swap space allocation when using older releases of Linux that don't use zram or in cases where zram has been disabled.
The documents I found for the Zram feature are inconsistent in terms of how zram is allocated with respect to RAM size, and the amount of space allocated to zram swap.
Due to the lack of authoritative documentation, I performed some experiments to empirically determine the algorithm used to allocate zram swap. I used my own physical and virtual systems for this. The results are interesting and do not match any documentation I've so far found.
The default size of zram is 8 GB on all systems large enough to support that, but it's typically reduced significantly on hosts with small amounts of RAM. On one virtual machine (VM) I use for testing, with access to 4 GB of RAM, the zram virtual swap space is allocated to 3.8 GB. One old Dell I have contains 8 GB of RAM, and the zram is set to 7.6 GB. When RAM is reduced to 2 GB, Zram is reduced to 1.9 GB.
All physical and virtual hosts I have with more than 8 GB of RAM show exactly 8 GB of zram. This includes my primary workstation with 64 GB of RAM and other hosts with 16 GB or 32 GB of RAM.
Based on these few data points, I can draw the conclusion that the current default settings are for 8 GB of zram at most, and for zram to be 95% of RAM on hosts with 8 GB or less.
I have read a number of articles that mention other sizes for zram swap, even up to 100% of RAM, but those all seem to be theoretical rather than reality.
Your distribution may be different, but here are the actual zram swap allocations for Fedora and similar distributions:
-
RAM ⇐ 8 GB: 0.95 × RAM
-
RAM > 8 GB: 8 GB
Be aware that the zram swap size algorithm is not based on any recommendations for the "best" swap size for any given real-world system or application. This zram swap allocation is a rather probabilistic approach to what should work well on a wide range of Linux hosts. However, the fact that the maximum zram swap size is configured for 8 GB and the fact that I have always recommended 8 GB as the maximum amount of traditional swap, I think I can say it's reflective of the optimum sizes for zram swap.
Managing zram swapZram defaults are stored in the /usr/lib/systemd/zram-generator.conf configuration file. The following is from one of my test VMs with 5097 GB of RAM allocated.
# cat /usr/lib/systemd/zram-generator.conf# This config file enables a /dev/zram0 device with the default settings:
# - size - same as available RAM or 8GB, whichever is less
# - compression - most likely lzo-rle
#
# To disable, uninstall zram-generator-defaults or create empty
# /etc/systemd/zram-generator.conf file.
[zram0]
zram-size = min(ram, 8192)
You can change the default Zram swap size in the last line of the zram-generator.conf configuration file. I recommend against doing that, unless you can definitively show a reason for doing so, and test your results once you make any changes. Like many other configuration defaults in Linux, the zram ones have been well-tested and are appropriate for most use cases.
Monitor zramThe zramctl utility can be used to view the current state of zram.
# zramctlNAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lzo-rle 4.8G 4K 80B 12K 4 [SWAP]
The traditional swapon command can also be used to view swap including zram used as swap:
# swapon --showNAME TYPE SIZE USED PRIO
/dev/zram0 partition 4.8G 0B 100
One thing to be aware of is that zramctl does not report on zram when it contains no data, so the results contain null output. Tools like lsblk, swapon, top, free, htop, and so on, do show zram even when it contains no data.
Deactivate zramThe swapoff -a command turns off zram swap as well as traditional HDD or SSD storage used as swap. The swapon -a command does not show zram when it is empty. Use zramctl /dev/zram0 instead.
# swapon --show# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 120G 0 disk
├─sda1 8:1 0 1G 0 part /boot/efi
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 118G 0 part
├─vg01-root 253:0 0 10G 0 lvm /
├─vg01-swap 253:1 0 3G 0 lvm [SWAP]
├─vg01-usr 253:1 0 30G 0 lvm /usr
├─vg01-home 253:2 0 10G 0 lvm /home
├─vg01-var 253:3 0 30G 0 lvm /var
└─vg01-tmp 253:4 0 10G 0 lvm /tmp
sr0 11:0 1 1024M 0 rom
zram0 252:0 0 0B 0 disk
# zramctl
#
# zramctl /dev/zram0
NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lzo-rle 0B 0B 0B 0B 4
Note that /dev/zram0 doesn't show up in these commands as swap space until it's being used for that purpose. This caused me some confusion until my experiments showed it to be the case.
Creating Zram SwapZram itself has been around for about 20 years, but has only been in use as swap space on some distributions for the last year or two. The current Linux installation on some or all of your hosts may not have been created with zram for swap. If that's the case, it can be easily remedied.
For Fedora 32, the last release prior to the default use of zram for swap, it only takes three easy commands.
First, verify the presence of the zram-swap.service file, installed as part of the zram RPM package.
# systemctl status zram-swap● zram-swap.service - Enable compressed swap in memory using zram
Loaded: loaded (/usr/lib/systemd/system/zram-swap.service; disabled; vendor preset: disabled)
Active: inactive (dead)
Next, install the zram-generator-defaults and zram-generator packages.
# dnf install zram-generator-defaults zram-generatorEnable and start the zram-swap service:
# systemctl enable zram-swap.service# systemctl start zram-swap.service
And then verify that zram0 exists, and is being used as swap space:
# lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 120G 0 disk
├─sda1 8:1 0 2G 0 part /boot
└─sda2 8:2 0 118G 0 part
├─vg01-root 253:0 0 10G 0 lvm /
├─vg01-swap 253:1 0 3G 0 lvm [SWAP]
├─vg01-usr 253:2 0 35G 0 lvm /usr
├─vg01-tmp 253:3 0 15G 0 lvm /tmp
├─vg01-var 253:4 0 35G 0 lvm /var
└─vg01-home 253:5 0 20G 0 lvm /home
sr0 11:0 1 1024M 0 rom
zram0 252:0 0 7.5G 0 disk [SWAP]Improve swap with zram
That's all there is to it. It was easy with Fedora. Different distributions will likely be just as easy, with some possible different details in the package names and commands. Give zram swap a try on your computer. In my next article, I'll demonstrate some further zram options.
Zram is a tool for creating an in-RAM compressed cache, specifically for use as swap space.
Image by:Internet Archive Book Images. Modified by Opensource.com. CC BY-SA 4.0
Linux Sysadmin What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.4 key differences between Twitter and Mastodon
Social media is not always sociable, and sometimes we need a sufficient impetus to change what we do and what we read. I began using Twitter as a replacement for my RSS reader in 2008, which revolutionized how I read and learned up to that point. Tweets from educators and free and open source (FOSS) advocates worldwide kept me informed and engaged in a learning network that was without equal. That's changed over the past half dozen years, and recently a change in ownership and the shaping of what I read was driven more by an algorithm than by my personal interests and choices. During a yearly meetup of correspondents and editors of Opensource.com a few years ago, Seth Kenlon suggested giving Mastodon a try. I joined Fosstodon in 2019. Fosstodon is a Mastodon instance for a community of like-minded people who enjoy free and open source software.
Mastodon vs TwitterChange is not easy. Being a creature of habit, I stayed with my old standby even though it was becoming increasingly tiresome. The threat of its sale in the spring of 2022 invited me to reconsider Fosstodon.
1. Favorite instead of likeThe Mastodon interface is similar to Twitter. Rather than "liking" a post, you "favorite" a post on Mastodon by clicking the star icon under the post content.
Image by:(Don Watkins, CC BY-SA 4.0)
2. Share a postRe-sharing on my old network is a "retweet," but on Mastodon, it's a "boost." You click the double-arrow icon under the post content to boost a post.
Image by:(Don Watkins, CC BY-SA 4.0)
3. Mastodon instancesBecause anyone can run a Mastodon instance, different instances not only have unique communities (like the ones that form around specific hashtags on Twitter, but Mastodon also has hashtags). Some have a unique set of rules. For instance, unlike my former social network, there were content moderation rules on Fosstodon that seemed strict initially. I made a post unrelated to FOSS software, and my post was removed. I was told it had been removed because I'd not issued a "content warning." That irked me, so I looked for another instance and found a couple more to my liking. One was Mastodon.social, and the other Scholar.social. The former is a general server with no expectation about what you will post. The latter was an instance dedicated to academics. In all cases, there are well-enforced codes of conduct.
Each instance has rules, and while they differ slightly in the description, they clearly spell out what is and is not acceptable behavior. Fosstodon published its code of conduct, which established the rules and expectations of behavior on the site.
More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources 4. Open source social networkingIf you want to run your own Mastodon instance or help develop one, you'll be happy to know that Mastodon is open source. It uses an AGPLv3 license, and its source code is available as a Git repository. The software provides a social network server that uses the ActivityPub protocol to communicate with other servers worldwide.
Mastodon is not a single site on the internet but a series of sites spanning the globe and communicating with each other. This federated network is referred to as the "fediverse." Unlike other social networks, where there's a single owner of the network, Mastodon and other ActivityPub sites are owned by anyone who runs a server.
From a user's perspective, this doesn't matter at first. You can sign up on any Mastodon instance and then connect to all other instances.
There is power to this distributed design, though. If you encounter an instance with a community producing content you'd rather not see, you can block either a single user from that instance or the whole instance.
In the past month, I've returned to Fosstodon primarily because open source is my passion. I enjoy sharing open source content on Fosstodon because the other users of Fosstodon are generally receptive to posts about free and open source software. When I have something to share that's not considered appropriate on Fosstodon, I share it on Scholar.social or Mastodon.social.
Not all instances have topics they focus on, and even those that do often use their topical interests as a guideline rather than grounds for strict removal of posts. If you have a particular interest, you might be able to find a community built around that topic, and you're likely to see that you have an instant audience. Of course, you'll still always be able to communicate with users of other instances, too.
Try MastodonMastodon is not a corporation. All of its instances are staffed and supported by each server's contributors. Some instances make it easy to support them with Patreon or PayPal.
I have found the fediverse a welcoming place that brings joy back into social networking. Have you joined Mastodon? What are your takeaways? Let us know in the comments.
Mastodon is not a corporation. All of its instances are staffed and supported by each server's contributors. Here are a few other advantages.
Image by:Opensource.com
Alternatives What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Audit your sharding database algorithm
Thanks to the ShardingSphere community's continuous review and feedback to develop features such as data sharding and read/write splitting, our team found that some users create many shards when using the data sharding feature.
In such cases, there can be 1,000 physical tables corresponding to a sharding logical table, which disturbs users.
For instance, a SELECT * FROM t_order statement will lead to a full-route, which is obviously not the case for OLTP. This SQL can be placed in another Proxy to avoid blocking other requests.
However, if users are not familiar with Proxy or how to write a where condition and don't know that sharding is not supported in this condition, a full-route is still required.
A full-route can lower the performance of Proxy and even result in the failure of a reasonable request. Imagine that there are 1,000 shards in a physical database. If they are executed in parallel, 1,000 connections are needed, and if in serial, the request can lead to a timeout. For this reason, community users asked whether the unreasonable request could be intercepted directly.
Our team considered the issue for a while. One option is to simply block the full-route operation. Doing so requires a check in the code and adding a switch to the configuration file. On the other hand, if the user later needs to set a table to read-only or requires the update operation to carry a limit, does that mean the code and configuration change again? This approach obviously goes against the pluggable logic of Proxy.
In response to the above problems, the recently released Apache ShardingSphere 5.2.0 provides users with auditing for the SQL sharding function. The audit can either be an interception operation or a statistical operation. Similar to the sharding and unique key generation algorithms, the audit algorithm is plugin-oriented, user-defined, and configurable.
[ Related read 5 new improvements in Apache ShardingSphere ]
Next, I will elaborate on the implementation logic for auditing data sharding with specific SQL examples.
Audit for sharding interfaceThe entrance to Apache ShardingSphere's audit is in the org.apache.shardingsphere.infra.executor.check.SQLCheckEngine class, which will invoke the check method of the SQLChecker interface. Currently, the ShardingSphere audit contains an audit for permission (verify username and password) and an audit for sharding.
This example focuses on the parent interface implemented in the ShardingAuditChecker of audit for sharding.
Image by:(Yacine Si Tayeb, CC BY-SA 4.0)
You can learn its working principles quickly by viewing the check code of org.apache.shardingsphere.sharding.checker.audit.ShardingAuditChecker.
public interface ShardingAuditAlgorithm extends ShardingSphereAlgorithm {/**
* Sharding audit algorithm SQL check.
*
* @param sqlStatementContext SQL statement context
* @param parameters SQL parameters
* @param grantee grantee
* @param database database
* @return SQL check result
*/
SQLCheckResult CHECK(SQLStatementContext<?> sqlStatementContext, List<Object> parameters, Grantee grantee, ShardingSphereDatabase DATABASE);
}
This method obtains the audit strategies of all the sharding tables involved and invokes the audit algorithms configured in each sharding table audit strategy. An exception is displayed to the user if an audit algorithm fails to pass.
Some users may wonder what disableAuditNames does here. The sharding audit also allows users to skip this process. In some cases, users may need to execute SQL that should have been blocked by the audit, and they are aware of the impact of this SQL.
Users can utilize the Hint: disableAuditNames to skip audit interception, which will be described with practical examples later. The Proxy Administrators can configure allowHintDisable to control whether to allow users to skip this process. The default value is true, indicating that a Hint-based skip is permitted.
More on data science What is data science? What is Python? How to become a data scientist Data scientist: A day in the life Use JupyterLab in the Red Hat OpenShift Data Science sandbox Whitepaper: Data-intensive intelligent applications in a hybrid cloud blueprint MariaDB and MySQL cheat sheet Latest data science articles Audit for sharding algorithmThe audit for sharding algorithm interface org.apache.shardingsphere.sharding.spi.ShardingAuditAlgorithm is inherited from SPI class ShardingSphereAlgorithm. It inherits type and props properties and defines its own check method. If you want to customize your audit algorithm, just implement the interface and add it to INF.services.
Image by:(Yacine Si Tayeb, CC BY-SA 4.0)
public interface ShardingAuditAlgorithm extends ShardingSphereAlgorithm {/**
* Sharding audit algorithm SQL check.
*
* @param sqlStatementContext SQL statement context
* @param parameters SQL parameters
* @param grantee grantee
* @param database database
* @return SQL check result
*/
SQLCheckResult CHECK(SQLStatementContext<?> sqlStatementContext, List<Object> parameters, Grantee grantee, ShardingSphereDatabase DATABASE);
}
Apache ShardingSphere implements a general audit for sharding algorithm org.apache.shardingsphere.sharding.algorithm.audit.DMLShardingConditionsShardingAuditAlgorithm, namely the above-mentioned SQL statement that intercepts the full-route.
The algorithm makes decisions by determining whether the sharding condition is null. Of course, it won't intercept broadcast tables and non-sharding tables.
public final class DMLShardingConditionsShardingAuditAlgorithm implements ShardingAuditAlgorithm {@Getter
private Properties props;
@Override
public void init(final Properties props) {
this.props = props;
}
@SuppressWarnings({"rawtypes", "unchecked"})
@Override
public SQLCheckResult CHECK(final SQLStatementContext<?> sqlStatementContext, final List<Object> parameters, final Grantee grantee, final ShardingSphereDatabase DATABASE) {
IF (sqlStatementContext.getSqlStatement() instanceof DMLStatement) {
ShardingRule rule = DATABASE.getRuleMetaData().getSingleRule(ShardingRule.class);
IF (rule.isAllBroadcastTables(sqlStatementContext.getTablesContext().getTableNames())
|| sqlStatementContext.getTablesContext().getTableNames().stream().noneMatch(rule::isShardingTable)) {
RETURN NEW SQLCheckResult(TRUE, "");
}
ShardingConditionEngine shardingConditionEngine = ShardingConditionEngineFactory.createShardingConditionEngine(sqlStatementContext, DATABASE, rule);
IF (shardingConditionEngine.createShardingConditions(sqlStatementContext, parameters).isEmpty()) {
RETURN NEW SQLCheckResult(FALSE, "Not allow DML operation without sharding conditions");
}
}
RETURN NEW SQLCheckResult(TRUE, "");
}
@Override
public String getType() {
RETURN "DML_SHARDING_CONDITIONS";
}
}
I'd like to introduce another audit for the sharding algorithm: LimitRequiredShardingAuditAlgorithm. This algorithm can intercept SQL without carrying a limit in the update and delete operations.
As this algorithm is less universal, it is not currently integrated into Apache ShardingSphere. As you can see, it is very easy to implement a custom algorithm, which is why the audit for sharding framework is needed. Thanks to its plugin-oriented architecture, ShardingSphere boasts great scalability.
public final class LimitRequiredShardingAuditAlgorithm implements ShardingAuditAlgorithm {@Getter
private Properties props;
@Override
public void init(final Properties props) {
this.props = props;
}
@SuppressWarnings({"rawtypes", "unchecked"})
@Override
public SQLCheckResult CHECK(final SQLStatementContext<?> sqlStatementContext, final List<Object> parameters, final Grantee grantee, final ShardingSphereDatabase DATABASE) {
IF (sqlStatementContext instanceof UpdateStatementContext && !((MySQLUpdateStatement) sqlStatementContext.getSqlStatement()).getLimit().isPresent()) {
RETURN NEW SQLCheckResult(FALSE, "Not allow update without limit");
}
IF (sqlStatementContext instanceof DeleteStatementContext && !((MySQLDeleteStatement) sqlStatementContext.getSqlStatement()).getLimit().isPresent()) {
RETURN NEW SQLCheckResult(FALSE, "Not allow delete without limit");
}
RETURN NEW SQLCheckResult(TRUE, "");
}
@Override
public String getType() {
RETURN "LIMIT_REQUIRED";
}
}Use audit for sharding
Audit for sharding requires you to configure an audit strategy for logical tables. To help you get started quickly, its configuration is the same as that of the sharding algorithm and the sharding key value generator.
There is an algorithm definition and strategy definition, and a default audit strategy is also supported. If the audit strategy is configured in the logical table, it affects only that logical table.
If defaultAuditStrategy is configured in the logical table, it takes effect for all the logical tables under the sharding rule. Auditors are similar to ShardingAlgorithms, auditStrategy to databaseStrategy, and defaultAuditStrategy to defaultDatabaseStrategy or defaultTableStrategy.
Please refer to the following example. Only the configuration of the audit for sharding is displayed. You must configure the sharding algorithm and data source yourself.
rules:- !SHARDING
TABLES:
t_order:
actualDataNodes: ds_${0..1}.t_order_${0..1}
auditStrategy:
auditorNames:
- sharding_key_required_auditor
allowHintDisable: TRUE
defaultAuditStrategy:
auditorNames:
- sharding_key_required_auditor
allowHintDisable: TRUE
auditors:
sharding_key_required_auditor:
TYPE: DML_SHARDING_CONDITIONS
Step 1: Execute a query operation. An error is displayed as the audit strategy for intercepting the full-database route is configured.
mysql> SELECT * FROM t_order;ERROR 13000 (44000): SQL CHECK failed, error message: NOT allow DML operation WITHOUT sharding conditions
Step 2: Add HINT. The name of the HINT is /* ShardingSphere hint: disableAuditNames */，and disableAuditNames is followed by the auditorsNames configured in the preceding command.
If multiple names exist, separate them with spaces such as/* ShardingSphere hint: disableAuditNames=auditName1 auditName2*/. After using HINT, you can see that the SQL operation is successfully executed.
mysql> /* ShardingSphere hint: disableAuditNames=sharding_key_required_auditor */ SELECT * FROM t_order;+----------+---------+------------+--------+
| order_id | user_id | address_id | STATUS |
+----------+---------+------------+--------+
| 30 | 20 | 10 | 20 |
| 32 | 22 | 10 | 20 |
+----------+---------+------------+--------+
2 ROWS IN SET (0.01 sec)
Note: HINT requires you to modify the server.yaml configuration of Proxy. In addition, if you are using MySQL terminal to connect to Proxy directly, you need to add the -c property—otherwise, HINT comments will be filtered out of the MySQL terminal and will not be parsed by Proxy on the backend.
rules:- !SQL_PARSER
sqlCommentParseEnabled: TRUE
sqlStatementCache:
initialCapacity: 2000
maximumSize: 65535
parseTreeCache:
initialCapacity: 128
maximumSize: 1024
props:
proxy-hint-enabled: TRUE
mysql -uroot -proot -h127.0.0.1 -P3307 -cDistSQL with audit for sharding
As you can see from the release notes, Apache ShardingSphere 5.2.0 supports the following DistSQL with audit for sharding function:
CREATE SHARDING AUDITORALTER SHARDING AUDITOR
SHOW SHARDING AUDIT ALGORITHMS
The following DistSQL will be supported IN future releases:
DROP SHARDING AUDITOR
SHOW UNUSED SHARDING AUDIT ALGORITHMS
CREATE SHARDING TABLE RULE # including AUDIT_STRATEGY
This post introduced how audit for sharding works with specific examples. I believe you already have a basic understanding of this function and can use it whenever you need or use a custom algorithm.
You are also welcome to submit general algorithms to the community. If you have any ideas to contribute or encounter issues with your ShardingSphere, feel free to post them on GitHub.
This article originally appeared on ShardingSphere 5.2.0: Audit for sharding intercepts unreasonable requests in multi-shards scenarios and is republished with permission.
This demonstration of ShardingSphere 5.2.0 elaborates on the implementation logic for auditing data sharding with specific SQL examples.
Image by:Image by Mapbox Uncharted ERG, CC-BY 3.0 US
Databases What to read next Improve your database knowledge with this MariaDB and MySQL cheat sheet This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Why sysadmins should choose Awesome window manager on Linux
Awesome is a window manager for the Linux desktop. A "window manager" is a graphical interface that primarily (if not literally) just manages the drawing and arrangement of windows. In practice, even the most rudimentary of window managers actually provides a little more than just the ability to draw a window. Most also provide a pop-up menu so you can launch an application, some provide a dock or panel so you can switch between different applications you have running. They stop short at providing desktop conveniences such as drawing a wallpaper in the background of your screen, mounting and unmounting devices, providing a system tray, and so on. A window manager assumes you can use other applications to build a desktop experience to your own liking, and so it focuses on managing windows. The Awesome window manager takes a "tiling" approach, meaning that each window you launch takes up a fraction of your desktop according to the number of windows you have open.
Image by:(Seth Kenlon, CC BY-SA 4.0)
My Linux desktop is the terminalWhen you're a systems administrator, you tend to spend a lot of time in a terminal window. It's a direct and efficient interface to your local machine, to remote machines, the network, the Internet, and everything else, so it's usually the easiest and most sensible way to do a lot of things to a lot of computers at once. And when you spend all day in a terminal, you understandably start to question whether you actually need a desktop at all.
More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articlesTo be perfectly honest, the answer's often no, at least for 80% of your tasks. The reality of modern computing, however, is that there are some applications that are just easier to use through a graphical interface. For instance, even though there are issue tracking systems, like the open source Bugzilla, that provide terminal commands as an interface, sometimes you're on a team that uses an issue tracker (usually it's not open source) that provides only a web application. Like it or not, you need a web browser to interact with the ticketing system. And even though you may use a perfectly reasonable markup language like AsciiDoc, you're probably sent a word processor document sometimes and, while you could use Pandoc to convert the document into AsciiDoc and then back into an office document, that risks losing something in translation. Like it or not, you need an office suite.
The bottom line is that, whether you like it or not, you need a desktop. Or at least a window manager.
Tiling windowsAwesome understands your plight. With Awesome, your "primary" desktop can be your terminal. When you first launch it, your terminal window is full screen, just like the text console you really want to be greeted with upon login. When you really need web browser, though, you can launch it and Awesome makes room for it by splitting your screen in half. Your terminal is on one side, the web browser's on the other.
If you need to open a third application, you can launch that and Awesome makes room for it by splitting your screen into thirds.
Image by:(Seth Kenlon, CC BY-SA 4.0)
When you're finished with an application, Awesome adjusts your layout again until, eventually, you're back to a full-screen terminal, just the way you like it.
Lua configurationAwesome uses the Lua scripting language for configuration. Lua has a similar philosophy to Awesome. It's a simple language that's pretty intuitive once you understand a few basic concepts.
The simplest concept, and yet the most important, is the Lua table construct. Lua stores information in what it calls a "table", and it means that most everything in Lua has a structured hierarchy. For instance, this creates a Lua table:
zombie = {} zombie.apocalypse = true zombie.defeat = falseNow when you need to know whether there's an active zombie apocalypse, you can "call" the zombie table and query the apocalypse value:
> print(zombie.apocalypse) trueBoth the apocalypse and the defeat values are "children" of the zombie table, which makes them each distinct from the apocalypse and defeat values of the alien table.
It's a simple system of data classification, and you see several tables used in the Awesome configuration:
-- Table of layouts awful.layout.layouts = { awful.layout.suit.floating, awful.layout.suit.tile, awful.layout.suit.tile.left, awful.layout.suit.tile.bottom, awful.layout.suit.tile.top, awful.layout.suit.fair, awful.layout.suit.fair.horizontal, awful.layout.suit.spiral, awful.layout.suit.spiral.dwindle, awful.layout.suit.max, awful.layout.suit.max.fullscreen, awful.layout.suit.magnifier, awful.layout.suit.corner.nw, }You may not know what options are available from reading the configuration file itself, but understanding that the options are grouped into Lua tables means you know what to look up in the Awesome documentation.
Of course, if you don't feel like reading through the documentation, you can also just comment options out and see what changes. A comment in Lua are two dashes, as in the first line of this snippet:
-- Create a launcher widget and a main menu myawesomemenu = { { "hotkeys", function() hotkeys_popup.show_help(nil, awful.screen.focused()) end }, { "manual", terminal .. " -e man awesome" }, { "edit config", editor_cmd .. " " .. awesome.conffile }, { "restart", awesome.restart }, { "quit", function() awesome.quit() end }, }Lua is a consistent and logical language. Any amount of Lua you pick up from configuring Awesome is Lua you can use for real life Lua scripting. If you're ready to move up from basic Bash scripting, you might consider Lua.
Adjusting tilesWhile you're working in a split screen in Awesome, you might find the need to adjust the proportions. To do this graphically, right-click on the intersection the windows you want to adjust, and then drag all window borders to suit your preference.
When you learn enough Lua to get really good at configuring Awesome, you can configure default preferences.
Floating windowsSome applications never make sense as tiled windows. The Awesome configuration provided by your Linux distribution probably has a few examples set:
-- Floating clients. class = { "Blueman-manager", "Kruler", "MessageWin", "Tor Browser", "Wpa_gui"}, },Assemble your own Linux desktopUsing a window manager instead of a desktop means you get to choose the components you use for everything else you want to do with your computer. You can launch KDE applications from the Plasma Desktop, or use bits and pieces of XFCE (such as the panel, the network manager, and more), or you can eschew the desktop model entirely and use a particularly robust file manager and the terminal commands you know and love.
The Awesome window manager takes a "tiling" approach, meaning that each window you launch takes up a fraction of your desktop according to the number of windows you have open.
Image by:WOCinTech Chat. Modified by Opensource.com. CC BY-SA 4.0
Linux What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Using Python in VS Code and Codium
Over the past couple of years, I have had the privilege of working with middle school children to introduce them to Python coding and the Raspberry Pi 400. It's been a lot of fun, and the Pi has been a great platform for the students and me. We've used Code with Mu and it's been quite successful. Our aptitude with Python has grown with experience, and so recently I started looking for ways to offer these students more.
I participated in a Python learning class, and during that class I got introduced to Microsoft's Visual Studio Code. I learned a lot in that class about how to set up a virtual environment for Python, and how to configure VS Code for Python programming. During that learning journey, I also got introduced to Codium, which is essentially VS Code without Microsoft's branding and telemetry.
If you're looking for a good, general-purpose, open source code editor with Python integration, then you might give Codium a try. Here's how I got Codium set up for Python on my Linux system.
Install or update Python on LinuxFirst, make sure you are running the latest version of Python. You can do this with your package manager. On Debian and Debian-based systems:
$ sudo apt install python3-pipOn Fedora, CentOS, Mageia, OpenMandriva, and similar:
$ sudo dnf update python3On some systems, you may also need to install the software to create Python virtual environments:
$ sudo apt install python3.10-venv More Python resources What is an IDE? Cheat sheet: Python 3.7 for beginners Top Python GUI frameworks Download: 7 essential PyPI libraries Red Hat Developers Latest Python articles Install CodiumNext, install Codium on your computer. On Linux, you can download a package and install it with your package manager, or use the Flatpak.
To launch Codium once it's installed, open your application or Activities menu and type "Code".
Install the VS Code Python extensionThere's nothing special about code. It's just plain text that gets interpreted by some other application, whether it's a compiler or a runtime. You can write Python code in Codium without special extensions. However, having a Python extension adds several conveniences.
Click on the File menu, select Preferences, and choose Extensions. In the Extensions panel, find the Python IntelliSense extension.
Image by:(Opensource.com, CC BY-SA 4.0)
You've got Python set up in Codium. All that's left to do is to put it to good use.
Setup a virtual environment for VS Code or CodiumYou can create a project directory and add it to Codium so that while you work, the files you create and save default to the active project directory. It's a fast way to stay organized, and it saves you from having to click around File Save and Open dialogues constantly.
When you create a virtual Python environment as a work folder, Codium (because you have the Python extension installed) detects it. When you activate a virtual environment folder as the active project directory, Codium automatically runs the activation code required to use the virtual environment.
To create a virtual environment for Python, open a terminal and type:
$ python3 -m venv ~/PythonCodingAdd a project directoryIn Codium, click on the File menu and choose Add Folder to Workspace. Open the virtual environment you've just set up (for me, that's /home/don/PythonCoding.)
Now you're ready to write some Python code! Create a new Python file in your workspace and insert some basic code. You may notice, as you type, that Codium helpfully suggests auto-completion for the Python modules the environment contains.
import sysprint ("Codium running Python " + sys.version)
Now click the Play button in the top right corner of the Codium window. This opens a console panel at the bottom of the window, displaying the output of your code:
(PythonCode) sh-5.1$ /home/bogus/PythonCode/bin/python /home/bogus/PythonCode/app.pyCodium running Python 3.10.6 (main…) [GCC 12.1.0]
(PythonCode) sh-5.1$
As you can see from this output, Codium is running within the PythonCode environment, and it's successfully run your Python code.
Codium and PythonUsing Codium for Python makes writing and running code easier than ever, but Python isn't the only language Codium supports. You can easily find and install other extensions from the Open VSX Registry, a vendor-neutral open source "marketplace" for VS Code extensions.
The Codium interface is more complicated than some basic editors, but it has what I'm looking for at this point in my learning journey. If you're looking to graduate to something professional, or you're looking to switch from your current editor to something new, then give Codium a try.
If you're looking for a good, general-purpose, open source code editor with Python integration, then you might give Codium a try.
Image by:WOCinTech Chat. Modified by Opensource.com. CC BY-SA 4.0
Python Linux What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.What stickers are on your laptop?
Having just switched work laptops last December, I realized how much I love applying stickers to my work machines. Sometimes the stickers are souvenirs from past events, others are from projects that I am passionate about, and some are just for fun!
Curious to know what others had on their laptops, I asked!
Accessibility advocate Image by:(AmyJune Hineline, CC BY-SA 4.0)
Here is my work laptop (Can you tell my roots are in Drupal). My favorite decal is the Druplicon with the wheelchair... When Drupal 8 came out, we took the logo and blended it with the wheelchair because the agency I worked with focused on accessibility.
Fresh Java Image by:(Alan Formy-Duval, CC BY-SA 4.0)
I have always had stickers on my computers and stuff since I was a kid. I think my favorite is either Tux or the Linux Inside. They are mostly field-relevant except for just a few. In the bottom-right corner, I have Java running on Tomcat (haha) - an area I spent much of my career doing.
Utilitarian purpose Image by:(Rikard Grossman-Nielsen, CC BY-SA 4.0)
Well, I don't have any stickers. However, I've glued Velcro bands on my two laptops to secure my external hard drive for when I'm gaming on the bus. I have also glued a lock notch on.
Maintain the look Image by:( John 'Warthog9' Hawley, CC BY-SA 4.0)
Not the most decorated laptop by far, but I like the collection (and you know, hard to get more without travel for a while!)
My favorite is the "last one to commit is the maintainer". It's a snarky comment on the state of maintainership, as well as a promise that the code will live on as a result.
Mostly it boils down to things I use or contribute to, think are meaningful, or just found the sticker awesome.
Window coveringI never put stickers on my laptop because it seems to me the only really cool stickers are the ones I don't have (said the grumpy old man.)
But the old homebrew computer my kids used to use in high school, a 3GHz Core Duo with 8 GB of memory, has an Open Mainframe sticker on it that I grabbed at the Linux Foundation Open Source Summit here in Vancouver a few years ago. I quite like that one.
And because in my life, the Control key lives next to the A, not down on the bottom row, I have a few keyboards around with a CTRL sticker on the CapsLock key and a CAPS sticker on the Control key, which work together with the swap Ctrl and CapsLock option in GNOME Tweak Tool.
Finally, I used to peel off Windows stickers, back when my only option was buying computers and paying the Windows tax, and put Linux sticker over the gummy patch. Same with keyboards that had the Windows logo on the Super key.
(Kristine Strange, CC BY-SA 4.0)
The Kanopi sticker is by far my favorite sticker. Not only is it shiny and iridescent, but it's a constant reminder of how amazing this company is to work for. They seriously put their employees first, and they're super mindful in selecting client projects that align with Kanopi's overall company mission and vision.
The Curt V8 sticker is in remembrance of a dear friend. He loved Fords and my husband loves Chevys. The constant fun rivalry resulted in randomly placed Ford and Chevy objects snuck into garages depending on whose house we were at. I smile every time I see this Ford emulated sticker on my laptop, since I live in a Chevy family.
The variety of stickers represents the family adventures that we have been on throughout the years. Date nights, friends, family road trips, scary hiking adventures (Angels Landing), and my youngest's drive to get a police sticker from every city and state.
(Cindy Williams, CC BY-SA 4.0)
The dragon is my college mascot. I also have some Gilmore Girls and coffee stickers.
Here’s a photo of my daughter’s door, filled with stickers I’ve brought back from conferences over the years.
Image by:(Cindy Williams, CC BY-SA 4.0)
Sticking with chickenThis is my not-work laptop. My work laptop is basically being covered in a honeycomb of hex-shaped stickers of our products, open source projects I use and support, and at least one Opensource.com hexagon. :)
I can’t pick a favorite, since they are all favorites, or I wouldn’t have them on the laptop that goes with me everywhere. I am overly fond of the chickens, the Raven, and Sergi the Cat with his knives.
Image by:(Kevin Sonney, CC BY-SA 4.0)
Image by:(Kevin Sonney, CC BY-SA 4.0)
Foodie funI used to load up my laptops with stickers. The one I bought last year filled up fast:
Image by:(DJ Billings, CC BY-SA 4.0)
My favorite is the cupcake & donut one because I illustrated it.
I just bought a System76 Darter Pro laptop and I love it. I got a bunch of really cool stickers with it, but I've been hesitant to put them on the laptop. I don't know why.
(Don Watkins, CC BY-SA 4.0)
I don’t put a lot of stickers on my laptops but I’ve got my two favorites on my current laptop, which is System76 Darter Pro.
Life's essentials Image by:(Katie Sanders, CC BY-SA 4.0)
I included my water bottle, too. I think I like those stickers even more.
Beer, dogs, music, croissants. What else could I need in life?
(Faye Polson, CC BY-SA 4.0)
My favorite sticker is yeet or be yeeted.
Garlic Image by:(Tiffany Bridges, CC BY-SA 4.0)
Most of the stickers are professional, but the Greetings from Hamunaptra, City of the Dead sticker is a subtle reference to one of my favorite movies, The Mummy (1999) starring Brendan Fraser and Rachel Weisz.
The flags and the Blackbeard’s Bar & Grill stickers are references to Our Flag Means Death, which I am completely obsessed with.
And the garlic is the Cosmic Garlic sticker of my friend's shop. Garlic is a folk remedy for all kinds of diseases, so it seemed like a good thing to put on a laptop during a pandemic.
(Seth Kenlon, CC BY-SA 4.0)
I usually cover my laptop with projects I use, contribute to, or admire. Statistically, my laptop should be layered with a lot more stickers by now. I haven't been to a tech conference in three years, so the pace has been slower than usual.
Decked out in Drupal Image by:(April Sides, CC BY-SA 4.0)
I add stickers that represent me in tech. So I include organizations I'm a part of, events I've attended, and projects I support.
It's always fun to see the Drupal bear on people’s laptops since I designed it.
Notice all of my stickers are on a laptop cover for preservability.
(Michelle Frechette', CC BY-SA 4.0)
My favorite is hard to pick, but probably the Michelle wapuu! She’s so me!
The stickers I press all the words and WordPress is my super power are from WordCamp Rochester, so those are near and dear to me.
Basically, I’ll add a sticker if I have a history with it (I spoke at the camp, for example), or I just like it!
An eye for art Image by:(Jonathan Daggerhart, CC BY-SA 4.0)
I heavily lean towards art stickers. Seeing art on my computer reminds me of the people I know and experiences I’ve had while using this computer.
My favorite is the sad-face Midsommar sticker that my partner gave me. After seeing the movie for the first time, we stood outside the theater discussing it for hours into the night. We still reference it to this day.
Custom skin Image by:(Sallie Goetsch, CC BY-SA 4.0)
I got a new travel laptop in 2019, and it remains pristine because I have not been to any events since then. My work laptop has a custom skin of the goddess Ereshkigal, after whom I named the computer.
—Sallie Goetsch
GNU Emacs Image by:(Sachin Patil, CC BY-SA 4.0)
Opensource.comAfter seeing everyone's responses, and maybe getting some cool stickers in the mail from a very brilliant community manager...
OK, OK, I give! AmyJune, Don, and Sachin convinced me to put ONE sticker on my laptop.
Here's a photo showing my laptop with its singular sticker:
Image by:(Chris Hermansen, CC BY-SA 4.0)
—Chris Hermansen
Stickers and open sourceYou don't have to adorn your computer with stickers. It's not required, and it certainly doesn't mean you love open source more or less than anybody else. But if you love an open source project, there's a good chance that it's got a sticker you can use to decorate your computer (or your door, water bottle, or USB microphone). Rest assured that if you love open source and you love stickers, there's a strong intersection between the two!
Your laptop is a blank canvas ready to be decorated with self-expression. What are your favorite stickers? Take our poll and then read what other open source enthusiasts had to say.
Image by:Opensource.com
Opensource.com community Hardware What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.What you actually need to know about open source to get started
So you want (or need) to figure out what "open source" really means. I'll cover the basics of open source, whether you're interested in contributing to a project or want to be in the loop at a new job where the term keeps getting thrown around.
Full disclosure: I am a person with little technical experience, working in the content-design fringes of a very technical open source environment. Given my background in marketing and communication, I felt like a fish out of water when I made this career switch. Git, data science, the ins and outs of software… It was, and still is a year later, a lot to comprehend.
But that's why I'm writing this piece. I want to help make open source a little less intimidating. After all, at the center of open source is a supportive learning community—built for everyone, technically experienced or not.
I'll start with the absolute basics.
What is open source?For the record, the industry definition of open source is available at the Open Source Initiative site.
However, the popular perception of "open source" software is usually that it doesn't cost anything, the source code is accessible, anyone can contribute to it, and you can redistribute it or do whatever else you want with it.
Some of that is true, and some of it plays into a few common misconceptions, one of which is cost.
More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources Open source costs $0Is it true? Usually, but not always. By nature of its code being publicly available, open source software can be obtained at no cost. However, for-profit companies do exist around open source projects. But if the software is available at no cost, how do open source companies even exist? How do they make money?
The concept of having a "free product" is counter-intuitive. But that's just the thing: A company doesn't have to sell software to profit from the management of products, storage of data, and customer support.
Many companies follow a subscription model, offering customer support in case of bugs or general confusion. Data storage isn't free, so that is another area where these companies can bring in income. In this regard, the "product" isn't the software; it's the benefit of a subscription.
- The source code is accessible: Is it true? Yes, always. This accessibility is a prerequisite for adopting the term "open source." The source code must be available to view, use, modify, and redistribute.
- You can do whatever you want with the code: Is it true? It depends. Subject to licensing terms, there are some limitations on how you can use code, but you can generally use it however you'd like. Whether that means tweaking a project to fit a specific need or using it as the basis for something else, open source software is yours, and everyone else's, to modify.
- Anyone can contribute to open source projects: Is it true? Yes - within limits. Anyone with the right skill set can contribute to open source. However, that doesn't mean all contributions are always accepted and implemented.
For example, say you're interested in a project where the end goal is a catalog of all the types of birds in the world. You're really into dinosaurs, specifically dinosaurs that may have eventually evolved into modern-day birds. So, you contribute entries for all of the most bird-like dinosaurs. The project owners could see this and think, "Sweet, those are some great prehistoric birds." However, they're also allowed to say, "Hmm, those dinosaurs are like birds, but they're technically not birds yet. They probably don't belong on Birdpedia."
Luckily, projects don't usually work under lawless conditions. Open source projects typically come with contribution guidelines and codes of conduct, so you don't have to worry about your additions flying off the rails.
Why open source?So, after all the contributions are made (if it's ever actually done), why would people give away their software for no cost? If so many people put their time and effort into creating something, why wouldn't they band together and slap a price tag on it?
This question comes with a lot of answers. Here are a few:
- Starting a business is hard, especially if the project you're working on doesn't form the strong foundation for a money machine. It can be easier to rally a bunch of like-minded people without commitments or the expectation of paychecks.
- Most open source communities consist of people interested in improving software or bringing it into existence but don't have the time or interest to commit to working full-time on a project. Sometimes open source represents passion projects, geek groups, and crowd-sourced solutions to annoying problems.
- The groups that form around open source projects of all sizes foster supportive communities where contributors and onlookers alike can practice their skills, improve software they regularly use, teach and learn from each other, and feel empowered to make their voices heard. Many open source communities are essentially hyper-focused online hobby clubs.
Now you may ask yourself, "But what do I do with this information? Can I contribute to open source projects? What if I'm not good enough yet?"
Never fear—even beginners are welcome to contribute to open source projects. It's a great way to hone your skills while working with a community towards a larger goal. And, as I talked about earlier, the worst that can happen is your changes aren't merged into Birdpedia (and that's because those product owners just can't see your vision of a Birdpedia where birds and their ancestors gleefully coexist in an online world of bird-related knowledge).
[ Related read Why do we contribute to open source software? ]
Do you have to know how to code to contribute to projects? Contrary to popular belief, no, you don't. Projects "take a village" to thrive, which means they need input from people of all different backgrounds. Visual designers, writers, marketers, reviewers, translators, subject matter enthusiasts, and even just users of the resulting product are all valuable contributors. Not only do they help build out and improve products, but they identify bugs, suggest improvements, spread the word about the project, and generally strengthen the community.
In short, no matter what your background or experience, if you're interested in open source or a specific project, you're nearly guaranteed to be welcomed with open arms.
Get started with open source nowStill not sure where to begin? Here are some ideas and resources to get you started:
- Up For Grabs is a "list of open source projects which have curated tasks specifically for new contributors." This is a great place to find an easy first PR opportunity, which is a great way to find out what kind of contributions you'll enjoy.
- Check out this list of beginner-friendly projects on GitHub.
- If you're still not feeling inspired, consider contributing to (or flying with) PatternFly, Red Hat's open design system.
A beginner's guide to open source explained in plain terms.
Image by:Opensource.com
What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Build your own SaaS on Linux with Vely
Vely combines high performance and the low footprint of C with the ease of use and improved safety of languages like PHP. It's free and open source software, licensed under GPLv3 and LGPL 3 for libraries, so you can even build commercial software with it.
Using Vely for SaaSYou can use Vely to create a multitenant web application that you can run on the Internet as Software-as-a-Service (SaaS). Each user has a completely separate data space from any other.
In this example web application, a user can sign up for a notebook service to create notes and then view and delete them. It demonstrates several technology integrations in just 310 lines of code across seven source files. The technologies include:
- MariaDB
- Web browser
- Apache
- Unix sockets
Here's how the application works from a user's perspective. A code walk-through follows the images.
The app allows a user to create a new login by specifying an email address and password. You can style these any way you like, such as with CSS:
Image by:(Sergio Mijatovic, CC BY-SA 4.0)
Verify the user's email:
Image by:(Sergio Mijatovic, CC BY-SA 4.0)
Each user logs in with their unique username and password:
Image by:(Sergio Mijatovic, CC BY-SA 4.0)
Once logged in, a user can add a note:
Image by:(Sergio Mijatovic, CC BY-SA 4.0)
A user can get a list of notes:
Image by:(Sergio Mijatovic, CC BY-SA 4.0)
The app asks for confirmation before deleting a note:
Image by:(Sergio Mijatovic, CC BY-SA 4.0)
After the user confirms, the note is deleted:
Image by:(Sergio Mijatovic, CC BY-SA 4.0)
Setup prerequisitesFollow the installation instructions on Vely.dev. It's a quick process that uses standard packaging tools, such as DNF, APT, Pacman, or Zypper.
Because they are part of this example, you must install Apache as a web server and MariaDB as a database.
After installing Vely, turn on syntax highlighting in Vim if you're using it:
vv -mGet the source codeThe source code for this demonstration SaaS app is part of the Vely installation. It's a good idea to create a separate source code directory for each application (and you can name it whatever you like). In this case, unpacking the source code does that for you:
$ tar xvf $(vv -o)/examples/multitenant_SaaS.tar.gz$ cd multitenant_SaaS
By default, the application is named multitenant_SaaS, but you can call it anything (if you do that, change it everywhere).
Set up the applicationThe very first step is to create an application. It's simple to do with Vely's vf utility:
$ sudo vf -i -u $(whoami) multitenant_SaaSThis command creates a new application home (/var/lib/vv/multitenant_SaaS) and performs the application setup for you. Mostly, that means creating various subdirectories in the home folder and assigning privileges. In this case, only the current user (the result of whoami) owns the directories, with 0700 privileges, which ensures that no one else has access to the files.
Set up the databaseBefore doing any coding, you need a place to store the information used by the application. First, create a MariaDB database called db_multitenant_SaaS, owned by the user vely with password your_password. You can change any of these values, but remember to change them everywhere during this example.
Logged in as root in the MySQL utility:
CREATE DATABASE IF NOT EXISTS db_multitenant_SaaS;CREATE USER IF NOT EXISTS vely IDENTIFIED BY 'your_password';
GRANT CREATE,ALTER,DROP,SELECT,INSERT,DELETE,UPDATE ON db_multitenant_SaaS.* TO vely;
Then create database objects (tables and records and so on) in the database:
USE db_multitenant_SaaS;SOURCE setup.sql;
exitConnect Vely to a database
To let Vely know where your database is and how to log into it, create a database config file named db_multitenant_SaaS. (This is the name used by the database statements in the source code, so if you change it, make sure you change it everywhere.)
Vely uses native MariaDB database connectivity, so you can specify any options that a given database lets you:
$ echo '[client]user=vely
password=your_password
database=db_multitenant_SaaS
protocol=TCP
host=127.0.0.1
port=3306' > db_multitenant_SaaSBuild the application
Use the vv utility to make the application, using the --db option to specify the MariaDB database and the database config file:
$ vv -q --db=mariadb:db_multitenant_SaaS Programming and development Red Hat Developers Blog Programming cheat sheets Try for free: Red Hat Learning Subscription eBook: An introduction to programming with Bash Bash shell scripting cheat sheet eBook: Modernizing Enterprise Java An open source developer's guide to building applications Start the application serverTo start the application server for your web application, use the vf FastCGI process manager. The application server uses a Unix socket to communicate with the web server (creating a reverse proxy):
$ vf -w 3 multitenant_SaaSThis starts three daemon processes to serve the incoming requests. You can also start an adaptive server that increases the number of processes to serve more requests and gradually reduce the number of processes when they're not needed:
$ vf multitenant_SaaSSee vf for more options to help you achieve the best performance.
When you need to stop your application server, use the -m quit option:
$ vf -m quit multitenant_SaaSSet up the web serverThis is a web application, so the application needs a web server. This example uses Apache by way of a Unix socket listener.
1. Set up ApacheTo configure Apache as a reverse proxy and connect your application to it, you need to enable FastCGI proxy support, which generally means using the proxy and proxy_fcgi modules.
For Fedora systems (or others, like Arch) enable the proxy and proxy_fcgi modules by adding (or uncommenting) the appropriate LoadModule directives in the /etc/httpd/conf/httpd.conf Apache configuration file.
For Debian, Ubuntu, and similar systems, enable the proxy and proxy_fcgi modules:
$ sudo a2enmod proxy$ sudo a2enmod proxy_fcgi
For OpenSUSE, add these lines to the end of /etc/apache2/httpd.conf:
LoadModule proxy_module modules/mod_proxy.soLoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so2. Configure Apache
Now you must add the proxy information to the Apache configuration file:
ProxyPass "/multitenant_SaaS" unix:///var/lib/vv/multitenant_SaaS/sock/sock|fcgi://localhost/multitenant_SaaSThe location of your configuration may vary, depending on your Linux distribution:
- Fedora, CentOS, Mageia, and Arch: /etc/httpd/conf/httpd.conf
- Debian, Ubuntu, Mint: /etc/apache2/apache2.conf
- OpenSUSE: /etc/apache2/httpd.conf
Finally, restart Apache. On Fedora and similar systems, as well as Arch Linux:
$ sudo systemctl restart httpdOn Debian and Debian-based systems, as well as OpenSUSE:
$ sudo systemctl restart apache2Set up local mailThis example uses email as a part of its function. If your server can already send email, you can skip this. Otherwise, you can use local mail (myuser@localhost) just to test it out. To do that, install Sendmail.
On Fedora and similar:
$ sudo dnf install sendmail$ sudo systemctl start sendmail
On Debian systems (like Ubuntu):
$ sudo apt install sendmail$ sudo systemctl start sendmail
When the application sends an email to a local user, such as OS_user@localhost, then you can verify that the email was sent by looking at /var/mail/ (the "mail spool").
Access the application server from the browserAssuming you're running the application locally, use http://127.0.0.1/multitenant_SaaS?req=notes&action=begin to access your application server from your web browser. If you're running this on a live server on the Internet, you may need to adjust your firewall settings to allow HTTP traffic.
Source codeThis example application contains seven source files. You can review the code yourself (remember, it's just 310 lines across these files), but here's an overview of each one.
SQL setup (setup.sql)The two tables created are:
- users: Information about each user. Each user in the users table has its own unique ID (userId column) along with other information such as email address and whether it's verified. There's also a hashed password. An actual password is never stored in plain text (or otherwise); a one-way hash is used to check the password.
- notes: Notes entered by the user. The notes table contains the notes, each along with userId column that states which user owns them. The userId column's value matches the namesake column from users table. This way, every note clearly belongs to a single user.
The file contents:
CREATE TABLE IF NOT EXISTS notes (dateOf datetime, noteId BIGINT AUTO_INCREMENT PRIMARY KEY, userId BIGINT, note VARCHAR(1000));CREATE TABLE IF NOT EXISTS users (userId BIGINT AUTO_INCREMENT PRIMARY KEY, email VARCHAR(100), hashed_pwd VARCHAR(100), verified SMALLINT, verify_token VARCHAR(30), SESSION VARCHAR(100));
CREATE UNIQUE INDEX IF NOT EXISTS users1 ON users (email);Run-time data (login.h)
To properly display the Login, Sign Up, and Logout links, you need some flags that are available anywhere in the application. Also, the application uses cookies to maintain a session, so this needs to be available anywhere, for example, to verify that the session is valid. Every request sent to the application is confirmed that way. Only requests that come with verifiable cookies are permitted.
So to that effect, you have a global_request_data type reqdata (request data) and in it there's sess_userId (ID of user) and sess_id (user's current session ID). You also have rather self-explanatory flags that help render pages:
#ifndef _VV_LOGIN#define _VV_LOGIN
typedef struct s_reqdata {
bool displayed_logout; // true if Logout link displayed
bool is_logged_in; // true if session verified logged-in
char *sess_userId; // user ID of current session
char *sess_id; // session ID
} reqdata;
void login_or_signup ();
#endifSession checking and session data (_before.vely)
Vely has a notion of a before_request_handler. The code you write executes before any other code that handles a request. To do this, all you need is to write this code in a file named _before.vely, and the rest is automatically handled.
Anything that a SaaS application does, such as handling requests sent to an application, must be validated for security. This way, the application knows whether the caller has the permissions needed to perform an action.
Checking for permission is done here in a before-request handler. That way, whatever other code you have handling a request, you already have the session information.
To keep session data (like session ID and user ID) available anywhere in your code, you use global_request_data. It's just a generic pointer (void*) to memory that any code that handles requests can access. This is perfect for handling sessions, as shown below:
#include "vely.h"#include "login.h"
// _before() is a before-request-handler. It always executes before
// any other code that handles a request. It's a good place for any
// kind of request-wide setting or data initialization
void _before() {
// Output HTTP header
out-header default
reqdata *rd; // this is global request data, see login.h
// allocate memory for global request data, will be automatically deallocated
// at the end of request
new-mem rd size sizeof(reqdata)
// initialize flags
rd->displayed_logout = false;
rd->is_logged_in = false;
// set the data we created to be global request data, accessible
// from any code that handles a request
set-req data rd
// check if session exists (based on cookies from the client)
// this executes before any other request-handling code, making it
// easier to just have session information ready
_check_session ();
}Checking if the session is valid (_check_session.vely)
One of the most important tasks in a multitenant SaaS application is to check (as soon as possible) if the session is valid by checking whether a user is logged in. It's done by getting the session ID and user ID cookies from the client (such as a web browser) and checking these against the database where sessions are stored:
#include "vely.h"#include "login.h"
// Check if session is valid
void _check_session () {
// Get global request data
reqdata *rd;
get-req data to rd
// Get cookies from user browser
get-cookie rd->sess_userId="sess_userId"
get-cookie rd->sess_id="sess_id"
if (rd->sess_id[0] != 0) {
// Check if session ID is correct for given user ID
char *email;
run-query @db_multitenant_SaaS = "select email from users where userId='%s' and session='%s'" output email : rd->sess_userId, rd->sess_id row-count define rcount
query-result email to email
end-query
if (rcount == 1) {
// if correct, set logged-in flag
rd->is_logged_in = true;
// if Logout link not display, then display it
if (rd->displayed_logout == false) {
@Hi <<p-out email>>! <a href="https://opensource.com/?req=login&action=logout">Logout</a><br/>
rd->displayed_logout = true;
}
} else rd->is_logged_in = false;
}
}Signing up, Logging in, Logging out (login.vely)
The basis of any multitenant system is the ability for a user to sign up, log in, and log out. Typically, signing up involves verifying the email address; more often than not, the same email address is used as a username. That's the case here.
There are several subrequests implemented here that are necessary to perform the functionality:
- When Signing Up a new user, display the HTML form to collect the information. The URL request signature for this is req=login&action=newuser.
- As a response to the Sign Up form, create a new user. The URL request signature is req=login&action=createuser. The input-param signal obtains an email and pwd POST form fields. The password value is a one-way hash, and an email verification token is created as a random five-digit number. These are inserted into the users table, creating a new user. A verification email is sent, and the user is prompted to read the email and enter the code.
- Verify the email by entering the verification code sent to that email. The URL request signature is req=login&action=verify.
- Display a Login form for the user to log in. The URL request signature is req=login (for instance, action is empty.)
- Log in by verifying the email address (username) and password. The URL request signature is req=login&action=login.
- Logout at the user's request. The URL request signature is req=login&action=logout.
- Landing page for the application. The URL request signature is req=login&action=begin.
- If the user is currently logged in, go to the application's landing page.
See examples of these below:
#include "vely.h"#include "login.h"
// Handle session maintenance, login, logout, session verification
// for any multitenant Cloud application
void login () {
// Get URL input parameter "action"
input-param action
// Get global request data, we record session information in it, so it's handy
reqdata *rd;
get-req data to rd
// If session is already established, the only reason why we won't proceed to
// application home is if we're logging out
if (rd->is_logged_in) {
if (strcmp(action, "logout")) {
_show_home();
exit-request
}
}
// Application screen to get started. Show links to login or signup and show
// home screen appropriate for this
if (!strcmp (action, "begin")) {
_show_home();
exit-request
// Start creating new user. Ask for email and password, then proceed to create user
// when this form is submitted.
} else if (!strcmp (action, "newuser")) {
@Create New User<hr/>
@<form action="https://opensource.com/?req=login" method="POST">
@<input name="action" type="hidden" value="createuser">
@<input name="email" type="text" value="" size="50" maxlength="50" required autofocus placeholder="Email">
@<input name="pwd" type="password" value="" size="50" maxlength="50" required placeholder="Password">
@<input type="submit" value="Sign Up">
@</form>
// Verify code sent to email by user. The code must match, thus verifying email address
} else if (!strcmp (action, "verify")) {
input-param code
input-param email
// Get verify token based on email
run-query @db_multitenant_SaaS = "select verify_token from users where email='%s'" output db_verify : email
query-result db_verify to define db_verify
// Compare token recorded in database with what user provided
if (!strcmp (code, db_verify)) {
@Your email has been verifed. Please <a href="https://opensource.com/?req=login">Login</a>.
// If matches, update user info to indicate it's verified
run-query @db_multitenant_SaaS no-loop = "update users set verified=1 where email='%s'" : email
exit-request
}
end-query
@Could not verify the code. Please try <a href="https://opensource.com/?req=login">again</a>.
exit-request
// Create user - this runs when user submits form with email and password to create a user
} else if (!strcmp (action, "createuser")) {
input-param email
input-param pwd
// create hashed (one-way) password
hash-string pwd to define hashed_pwd
// generate random 5 digit string for verify code
random-string to define verify length 5 number
// create user: insert email, hashed password, verification token. Current verify status is 0, or not verified
begin-transaction @db_multitenant_SaaS
run-query @db_multitenant_SaaS no-loop = "insert into users (email, hashed_pwd, verified, verify_token, session) values ('%s', '%s', '0', '%s', '')" : email, hashed_pwd, verify affected-rows define arows error define err on-error-continue
if (strcmp (err, "0") || arows != 1) {
// if cannot add user, it probably doesn't exist. Either way, we can't proceed.
login_or_signup();
@User with this email already exists.
rollback-transaction @db_multitenant_SaaS
} else {
// Create email with verification code and email it to user
write-string define msg
@From: vely@vely.dev
@To: <<p-out email>>
@Subject: verify your account
@
@Your verification code is: <<p-out verify>>
end-write-string
exec-program "/usr/sbin/sendmail" args "-i", "-t" input msg status define st
if (st != 0) {
@Could not send email to <<p-out email>>, code is <<p-out verify>>
rollback-transaction @db_multitenant_SaaS
exit-request
}
commit-transaction @db_multitenant_SaaS
// Inform the user to go check email and enter verification code
@Please check your email and enter verification code here:
@<form action="https://opensource.com/?req=login" method="POST">
@<input name="action" type="hidden" value="verify" size="50" maxlength="50">
@<input name="email" type="hidden" value="<<p-out email>>">
@<input name="code" type="text" value="" size="50" maxlength="50" required autofocus placeholder="Verification code">
@<button type="submit">Verify</button>
@</form>
}
// This runs when logged-in user logs out.
} else if (!strcmp (action, "logout")) {
// Update user table to wipe out session, meaning no such user is logged in
if (rd->is_logged_in) {
run-query @db_multitenant_SaaS = "update users set session='' where userId='%s'" : rd->sess_userId no-loop affected-rows define arows
if (arows == 1) {
rd->is_logged_in = false; // indicate user not logged in
@You have been logged out.<hr/>
}
}
_show_home();
// Login: this runs when user enters user name and password
} else if (!strcmp (action, "login")) {
input-param pwd
input-param email
// create one-way hash with the intention of comparing with user table - password is NEVER recorded
hash-string pwd to define hashed_pwd
// create random 30-long string for session ID
random-string to rd->sess_id length 30
// Check if user name and hashed password match
run-query @db_multitenant_SaaS = "select userId from users where email='%s' and hashed_pwd='%s'" output sess_userId : email, hashed_pwd
query-result sess_userId to rd->sess_userId
// If match, update user table with session ID
run-query @db_multitenant_SaaS no-loop = "update users set session='%s' where userId='%s'" : rd->sess_id, rd->sess_userId affected-rows define arows
if (arows != 1) {
@Could not create a session. Please try again. <<.login_or_signup();>> <hr/>
exit-request
}
// Set user ID and session ID as cookies. User's browser will return those to us with every request
set-cookie "sess_userId" = rd->sess_userId
set-cookie "sess_id" = rd->sess_id
// Display home, make sure session is correct first and set flags
_check_session();
_show_home();
exit-request
end-query
@Email or password are not correct. <<.login_or_signup();>><hr/>
// Login screen, asks user to enter user name and password
} else if (!strcmp (action, "")) {
login_or_signup();
@Please Login:<hr/>
@<form action="https://opensource.com/?req=login" method="POST">
@<input name="action" type="hidden" value="login" size="50" maxlength="50">
@<input name="email" type="text" value="" size="50" maxlength="50" required autofocus placeholder="Email">
@<input name="pwd" type="password" value="" size="50" maxlength="50" required placeholder="Password">
@<button type="submit">Go</button>
@</form>
}
}
// Display Login or Sign Up links
void login_or_signup() {
@<a href="https://opensource.com/?req=login">Login</a> & & <a href="https://opensource.com/?req=login&action=newuser">Sign Up</a><hr/>
}General-purpose application (_show_home.vely)
With this tutorial, you can create any multitenant SaaS application you want. The multitenant-processing module above (login.vely) calls the _show_home() function, which can house any code of yours. This example code shows the Notes application, but it could be anything. The _show_home() function calls any code you wish and is a general-purpose multitenant application plug-in:
#include "vely.h"void _show_home() {
notes();
exit-request
}Notes application (notes.vely)
The application is able to add, list, and delete any given note:
#include "vely.h"#include "login.h"
// Notes application in a multitenant Cloud
void notes () {
// get global request data
reqdata *rd;
get-req data to rd
// If session invalid, display Login or Signup
if (!rd->is_logged_in) {
login_or_signup();
}
// Greet the user
@<h1>Welcome to Notes!</h1><hr/>
// If not logged in, exit - this ensures security verification of user's identity
if (!rd->is_logged_in) {
exit-request
}
// Get URL parameter that tells Notes what to do
input-param subreq
// Display actions that Notes can do (add or list notes)
@<a href="https://opensource.com/?req=notes&subreq=add">Add Note</a> <a href="https://opensource.com/?req=notes&subreq=list">List Notes</a><hr/>
// List all notes for this user
if (!strcmp (subreq, "list")) {
// select notes for this user ONLY
run-query @db_multitenant_SaaS = "select dateOf, note, noteId from notes where userId='%s' order by dateOf desc" : rd->sess_userId output dateOf, note, noteId
query-result dateOf to define dateOf
query-result note to define note
query-result noteId to define noteId
// change new lines to <br/> with fast cached Regex
match-regex "\n" in note replace-with "<br/>\n" result define with_breaks status define st cache
if (st == 0) with_breaks = note; // nothing was found/replaced, just use original
// Display a note
@Date: <<p-out dateOf>> (<a href="https://opensource.com/?req=notes&subreq=delete_note_ask¬e_id=%3C%3Cp-out%20noteId%3E%3E">delete note</a>)<br/>
@Note: <<p-out with_breaks>><br/>
@<hr/>
end-query
}
// Ask to delete a note
else if (!strcmp (subreq, "delete_note_ask")) {
input-param note_id
@Are you sure you want to delete a note? Use Back button to go back, or <a href="https://opensource.com/?req=notes&subreq=delete_note¬e_id=%3C%3Cp-out%20note_id%3E%3E">delete note now</a>.
}
// Delete a note
else if (!strcmp (subreq, "delete_note")) {
input-param note_id
// Delete note
run-query @db_multitenant_SaaS = "delete from notes where noteId='%s' and userId='%s'" : note_id, rd->sess_userId affected-rows define arows no-loop error define errnote
// Inform user of status
if (arows == 1) {
@Note deleted
} else {
@Could not delete note (<<p-out errnote>>)
}
}
// Add a note
else if (!strcmp (subreq, "add_note")) {
// Get URL POST data from note form
input-param note
// Insert note under this user's ID
run-query @db_multitenant_SaaS = "insert into notes (dateOf, userId, note) values (now(), '%s', '%s')" : rd->sess_userId, note affected-rows define arows no-loop error define errnote
// Inform user of status
if (arows == 1) {
@Note added
} else {
@Could not add note (<<p-out errnote>>)
}
}
// Display an HTML form to collect a note, and send it back here (with subreq="add_note" URL param)
else if (!strcmp (subreq, "add")) {
@Add New Note
@<form action="https://opensource.com/?req=notes" method="POST">
@<input name="subreq" type="hidden" value="add_note">
@<textarea name="note" rows="5" cols="50" required autofocus placeholder="Enter Note"></textarea>
@<button type="submit">Create</button>
@</form>
}
}SaaS with C performance
Vely makes it possible to leverage the power of C in your web applications. A multitenant SaaS application is a prime example of a use case that benefits from that. Take a look at the code examples, write some code, and give Vely a try.
Vely makes it possible to leverage the power of C in your web applications.
Image by:Opensource.com
Databases Programming Linux What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Fix scanned images with ImageMagick
Years ago while rummaging through the contents of a shelf in a used bookstore, I happened upon a booklet titled "UNIX System Command Summary for Berkeley 4.2 & 4.3 BSD," published by Specialized Systems Consultants. I bought it as a curiosity item because it was nearly 20 years old yet still largely applicable to modern Linux and BSD.
That amused me then and now. A booklet written in 1986 was still largely relevant in 2016, while books on the same shelf about a proprietary OS weren't worth the paper they were printed on. (Think about it: What technology do you think is going to survive a zombie apocalypse?) I've had the booklet on my own bookshelf for several years now, but it occurred to me that it's probably worth doing a little digital preservation of this artifact, so I decided to scan the booklet to create a CBZ ebook.
Scanning was easy, albeit time-consuming, with Skanlite. After I was finished, however, I discovered that some pages weren't quite level.
Image by:(Seth Kenlon, CC BY-SA 4.0)
In printing, this is called a registration problem, meaning that the position of what's being printed isn't correctly orientated on the page.
More open source alternatives Open source project management tools Trello alternatives Linux video editors Open source alternatives to Photoshop List of open source alternatives Latest articles about open source alternatives ImageMagickImageMagick is a non-interactive terminal-based graphics editor. It might seem counterintuitive to try to edit a graphic in a graphic-less environment like a text-only terminal, but it's actually very common. For instance, when you upload an image to use as a profile picture to a web application, it's likely that a script on the application's server processes your image using ImageMagick or its libraries. The advantage of a non-interactive editor is that you can formulate what needs to be done to a sample image, then apply those effects to hundreds of other images at the press of a button.
ImageMagick is generally just as capable as any graphics editor, as long as you take the time to uncover its many functions and how to combine them to achieve the desired effects. In this case, I want to rotate pages that are askew. After searching through ImageMagick's documentation, I discovered that the ImageMagick term for the solution I needed was called deskew. Aligning your terminology with somebody else's terminology is a challenge in anything that you don't already know, so when you approach ImageMagick (or anything), keep in mind that the word you've decided describes a problem or solution may not be the same word used by someone else.
To deskew an image with crooked text using ImageMagick:
$ convert page_0052.webp -deskew 25% fix_0052.webpThe -deskew option represents the threshold of acceptable skew. A skew is determined by tracing peaks and valleys of objects that appear to be letters. Depending on how crooked your scan is, you may need more or less than 25% threshold. I've gone as high as 80%, and so far nothing under 25% has had an effect.
Here's the result:
Image by:(Seth Kenlon, CC BY-SA 4.0)
Fixed! Applying this to the remaining 55 pages of the document fixed skewed pages while doing nothing to pages that were already straight. In other words, it was safe to run this command on pages that needed no adjustment, thanks to my threshold setting.
Cropping an image with ImageMagickAfter correcting for a skew, and because I scanned more of each page than necessary anyway to prevent accidentally cutting off words, I decided that it made sense to crop my corrected pages. I was happy to keep some space around the margins, but not quite as much as I had. I use the crop function of ImageMagick often enough for images on this very website, so I was familiar with the option. However, I needed to determine how to crop each page.
First, I needed the size of the image:
$ identify fixed_0052.webpWEBP 1128x2593 1128x2593+0+0 8-bit sRGB 114732B 0.020u 0:00.021
Knowing the size, I was able to make some estimations about how many pixels I could stand to lose. After a few trial runs, I came up with this:
convert fix_0052.webp -gravity Center -crop 950x2450+0+0 crop_0052.webpThis isn't an exact fit, but it proved important when I applied it to other images in the booklet. The pages varied in content and scanner placement here and there, so I was happy to give each one a little breathing room.
Here's the corrected and cropped image:
Image by:(Seth Kenlon, CC BY-SA 4.0)
Batch image editing with open sourceThe beauty of ImageMagick is that once you've figured out the formula for fixing your image, you can apply that fix to all images requiring the same fix. I do this with GNU Parallel, which uses all my CPU cores to finish image correction across hundreds of pages. It doesn't take long, and the results speak for themselves. More importantly, I've got a digital archive of a fun artifact of UNIX history.
It's easy to correct images, even in batches, with this open source tool.
Image by:Ray Smith
Tools Art and design What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 5014 points MinnesotaJim Hall is an open source software advocate and developer, best known for usability testing in GNOME and as the founder + project coordinator of FreeDOS. At work, Jim is CEO of Hallmentum, an IT executive consulting company that provides hands-on IT Leadership training, workshops, and coaching.
| Follow jimfhall | Connect jimfhall User Attributes Correspondent Open Sourcerer People's Choice Award People's Choice Award 2018 Author Correspondent Contributor Club Register or Login to post a comment.How to iterate over tables in Lua
In the Lua programming language, an array is called a table. A table is used in Lua to store data. If you're storing a lot of data in a structured way, it's useful to know your options for retrieving that data when you need it.
Programming and development Red Hat Developers Blog Programming cheat sheets Try for free: Red Hat Learning Subscription eBook: An introduction to programming with Bash Bash shell scripting cheat sheet eBook: Modernizing Enterprise Java An open source developer's guide to building applications Creating a table in LuaTo create a table in Lua, you instantiate the table with an arbitrary name:
mytable = {}There are different ways you can structure your data in a table. You could fill it with values, essentially creating a list (called a list in some languages):
mytable = { 'zombie', 'apocalypse' }Or you could create an associated array (called a map or dictionary in some languages). You can add arbitrary keys to the table using dot notation. You can also add a value to that key the same way you add a value to a variable:
myarray = {}myarray.baz = 'happy'
myarray.qux = 'halloween'
You can add verification with the assert() function:
assert(myarray.baz == 'happy', 'unexpected value in myarray.baz')assert(myarray.qux == 'halloween', 'unexpected value in myarray.qux')
You now have two tables: a list-style mytable and an associative array-style myarray.
Iterating over a table with pairsLua's pairs() function extracts key and value pairs from a table.
print('pairs of myarray:')for k,v in pairs(myarray) do
print(k,v)
end
Here's the output:
pairs of myarray:baz happy
qux halloween
If there are no keys in a table, Lua uses an index. For instance, the mytable table contains the values zombie and apocalypse. It contains no keys, but Lua can improvise:
print('pairs of mytable:')for k,v in pairs(mytable) do
print(k,v)
end
Here's the output:
1 zombie2 apocalypseIterating over a table with ipairs
To account for the fact that tables without keys are common, Lua also provides the ipairs function. This function extracts the index and the value:
print('ipairs of mytable:')for i,v in ipairs(mytable) do
print(i,v)
end
The output is, in this case, the same as the output of pairs:
1 zombie2 apocalypse
However, watch what happens when you add a key and value pair to mytable:
mytable.surprise = 'this value has a key'print('ipairs of mytable:')
for i,v in ipairs(mytable) do
print(i,v)
end
Lua ignores the key and value because ipairs retrieves only indexed entries:
1 zombie2 apocalypse
The key and value pair, however, have been stored in the table:
print('pairs of mytable:')for k,v in ipairs(mytable) do
print(k,v)
end
The output:
1 zombie2 apocalypse
surprise this value has a keyRetrieving arbitrary values
You don't have to iterate over a table to get data out of it. You can call arbitrary data by either index or key:
print('call by index:')print(mytable[2])
print(mytable[1])
print(myarray[2])
print(myarray[1])
print('call by key:')
print(myarray['qux'])
print(myarray['baz'])
print(mytable['surprise'])
The output:
call by index:apocalypse
zombie
nil
nil
call by key:
halloween
happy
this value has a keyData structures
Sometimes using a Lua table makes a lot more sense than trying to keep track of dozens of individual variables. Once you understand how to structure and retrieve data in a language, you're empowered to generate complex data in an organized and safe way.
Create structure that makes it easier to find stored data.
Image by:Opensource.com
Programming What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Is Lua worth learning?
Lua is a scripting language used for procedural programming, functional programming, and even object-oriented programming. It uses a C-like syntax, but is dynamically typed, features automatic memory management and garbage collection, and runs by interpreting bytecode with a register-based virtual machine. This makes it a great language for beginners, but also a powerful tool for experienced programmers.
Lua has been somewhat eclipsed from the public view by languages like Python and JavaScript, but Lua has several advantages that make it popular in some major software projects. Lua is easily embedded within other languages, meaning that you can include Lua files in the code base of something written in (for instance) Java and it runs as if it were native Java code. It sounds like magic, but of course there are projects like luaj working to make it possible, and it's only possible because Lua is designed for it. It's partly because of this flexibility that you're likely to find Lua as the scripting language for video games, graphic applications, and more.
As with anything, it takes time to perfect, but Lua is easy (and fun) to learn. It's a consistent language, a friendly language with useful error messages, and there's lots of great support online. Ready to get started?
Installing LuaOn Linux, you can install Lua using your distribution's package manager. For instance, on Fedora, CentOS, Mageia, OpenMandriva, and similar distributions:
$ sudo dnf install luaOn Debian and Debian-based systems:
$ sudo apt install luaFor Mac, you can use MacPorts or Homebrew.
$ sudo port install luaFor Windows, install Lua using Chocolatey.
To test Lua in an interactive interpreter, type lua in a terminal.
FunctionsAs with many programming languages, Lua syntax generally involves a built-in function or keyword, followed by an argument. For instance, the print function displays any argument you provide to it.
$ lua Lua 5.4.2 Copyright (C) 1994-2020 Lua.org, PUC-Rio > print('hello') helloLua's string library can manipulate words (called "strings" in programming.) For instance, to count the letters in a string, you use the len function of the string library:
> string.len('hello') 5VariablesA variable allows you to create a special place in your computer's memory for temporary data. You can create variables in Lua by inventing a name for your variable, and then putting some data into it.
> foo = "hello world" > print(foo) hello world > bar = 1+2 > print(bar) 3 Programming and development Red Hat Developers Blog Programming cheat sheets Try for free: Red Hat Learning Subscription eBook: An introduction to programming with Bash Bash shell scripting cheat sheet eBook: Modernizing Enterprise Java An open source developer's guide to building applications TablesSecond only to the popularity of variables in programming is the popularity of arrays. The word "array" literally means an arrangement, and that's all a programming array is. It's a specific arrangement of data, and because there is an arrangement, an array has the advantage of being structured. An array is often used to perform essentially the same purpose as a variable, except that an array can enforce an order to its data. In Lua, an array is called a table.
Creating a table is like creating a variable, except that you set its initial content to two braces ({}):
> mytable = {}When you add data to a table, it's also a lot like creating a variable, except that your variable name always begins with the name of the table, and is separated with a dot:
> mytable.foo = "hello world" > mytable.bar = 1+2 > print(mytable.foo) hello world > print(mytable.bar) 3Scripting with LuaRunning Lua in the terminal is great for getting instant feedback, but it's more useful to run Lua as a script. A Lua script is just a text file containing Lua code, which the Lua command can interpret and execute.
The eternal question, when just starting to learn a programming language, is how you're supposed to know what to write. This article has provided a good start, but so far you only know two or three Lua functions. The key, of course, is in documentation. The Lua language isn't that complex, and it's very reasonable to refer to the Lua documentation site for a list of keywords and functions.
Here's a practice problem.
Suppose you want to write a Lua script that counts words in a sentence. As with many programming challenges, there are many ways to go about this, but say the first relevant function you find in the Lua docs is string.gmatch, which can search for a specific character in a string. Words are usually separated by an empty space, so you decide that counting spaces + 1 ought to render a reasonably accurate count of the words they're separating.
Here's the code for that function:
function wc(words,delimiter) count=1 for w in string.gmatch(words, delimiter) do count = count + 1 end return count endThese are the components of that sample code:
-
function: A keyword declaring the start of a function. A custom function works basically the same way as built-in functions (like print and string.len.)
-
words and delimiter: Arguments required for the function to run. In the statement print('hello'), the word hello is an argument.
-
counter: A variable set to 1.
-
for: A loop using the string.gmatch function as it iterates over the words you've input into the function, and searches for the delimiter you've input.
-
count = count +1: For each delimiter found, the value of count is re-set to its current value plus 1.
-
end: A keyword ending the for loop.
-
return count: This function outputs (or returns) the contents of the count variable.
-
end: A keyword ending the function.
Now that you've created a function all your own, you can use it. That's an important thing to remember about a function. It doesn't run on its own. It waits for you to call it in your code.
Type this sample code into a text file and save it as words.lua:
function wc(words,delimiter) count=1 for w in string.gmatch(words, delimiter) do count = count + 1 end return count end result = wc('zombie apocalypse', ' ') print(result) result = wc('ice cream sandwich', ' ') print(result) result = wc('can you find the bug? ', ' ') print(result)You've just created a Lua script. You can run it with Lua. Can you find the problem with this method of counting words?
$ lua ./words.lua 2 3 6You might notice that the count is incorrect for the final phrase because there's a trailing space in the argument. Lua correctly detected the space and tallied it into count, but the word count is incorrect because that particular space happens not to delimit a word. I leave it to you to solve that problem, or to find other ways in which this method isn't ideal. There's a lot of rumination in programming. Sometimes it's purely academic, and other times it's a question of whether an application works at all.
Learning LuaLua is a fun and robust language, with progressive improvements made with each release, and an ever-growing developer base. You can use Lua as a utilitarian language for personal scripts, or to advance your career, or just as an experiment in learning a new language. Give it a try, and see what Lua brings to the table.
Lua is a fun and robust language, with progressive improvements made with each release, and an ever-growing developer base. Discover all of its possibilities.
Image by:Ray Smith
Programming What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.How open source weaves connections between countries
Long ago, when I was a tech administrator for Arlington Public Schools, I received an unexpected email saying, "Our school in India loves your children's stories, so we've printed them all out in braille."
Apparently, the school had found my collection of children's stories on the web. I spent a moment visualizing those students so far away, enjoying my stories on those printed pages of braille. The email's sender might not have realized how meaningful their email was to me, but it's stayed with me for two decades.
Tell collaborative stories with open sourceThat experience got me thinking about how technology can weave new connections between people in different countries. While surfing the web recently, I encountered an Inkscape enthusiast in Romania. His talent with the software was obvious.
You might not think of Romania as a tech hub, but people have all kinds of tech talent there. Talent exists everywhere. Open source, accessible to anyone who downloads it, can serve as a connector between cultures and countries. This enthusiast was having a fabulous time honing his skills using Inkscape, one of today's premier illustration tools—and it doesn't cost anything. He and I had something in common—something that could foster a friendship.
I then became curious whether the Opensource.com website gets regular web visits from Romania. A quick email to the staff confirmed that, yes, their website logs show regular visits from Romania. In fact, Opensource.com gets more monthly web visits from Romania than from Denmark, Norway, and New Zealand. Would you be surprised to learn that Opensource.com receives a lot of web traffic from Indonesia? I find that statistic delightful because it points towards a smaller world—a more inclusive, multicultural world.
My thoughts about open source as a global connector awakened again last month when I found out about an open source, web-based motion graphics editor named Motionity. The software was created by a software developer in Great Britain and she describes it as "a mix of Canva and Adobe After Effects."
This application is immensely exciting. I can envision my lighthearted, whimsical story, Ellen the Eagle Needs Glasses, drawn in Inkscape and brought to life using Motionity. To honor AlyssaX's programming work, we must find ways to use her software that showcase what it can do. If you're an illustrator or animator, get in touch with me—even better, surprise me with your finished creative project created using Inkscape and Motionity.
Explore Motionity for whatever uses you might have. Surprise AlyssaX with your best creative work. I want AlyssaX to experience the same feeling I had when I received that one-sentence email from India.
(Our language does not currently have a word for that kind of experience, but maybe we ought to coin a word. If you coin such a word, please tell me.)
More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources Forge global friendshipsFifteen years ago, I attended an international conference on wireless community networks. At that conference, I participated in a session explaining the workings of Serbia Wireless, a nonprofit grassroots community wireless network. One of the young presenters was from Serbia, and another was from Bosnia. I quickly understood its significance when I heard what they were doing with their homebrew wireless community network. These two young people were using open source hardware and open source software to blur the national boundary lines between the two previously warring nations. I honestly wanted to jump up and cheer when their motivation became apparent. After their session, I grabbed one of the presenters, Bogdan Tancic, for this six-minute video interview. Listen carefully to what he has to say.
Make connectionsIf you're looking for something meaningful to do in your life, find ways of connecting with people in different countries. Help these individuals cherish their cross-national friendships. If these new connections are grown repeatedly, the world would be an immeasurably better place. Open source is an outstanding tool to make that happen.
What are your own cross-national open source stories? Share your stories right here on Opensource.com.
If global friendships are grown repeatedly, the world would be an immeasurably better place. Open source is an outstanding tool to make that happen.
Image by:Opensource.com
Accessibility What to read next New open source tool catalogs African language resources Global communication in open source projects This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Achieve smart home interoperability with open source technology
Advertisers sell the myth of the "Smart Home" as a magical world where you automate everything with ease to simplify your life and free your mind of trivial tasks. It looks great on paper and TV, but so far, it's mainly been a battle of technologies and manufacturers.
Suppose you want to add a presence sensor from manufacturer A to your existing hub from manufacturer B so you can switch ON your light from manufacturer B. Or perhaps you want to switch OFF your air conditioner when your door is open, but the door sensor is not from the air conditioning manufacturer. Will it work?
There's a significant chance that the devices you want to link to one another don't talk the same "language." You might research compatibility for weeks before deciding to take your chances. It may or may not work. There are so many technologies, protocols, APIs, and cloud connectors involved that you just can't be sure until you spend a lot of money trying them. So far, the "Smart Home" has been an over-complex world in which every vendor tries to push its own technology, protocol, and stack at the cost of consumer freedom, time, effort, and expense.
Image by:(Florian Robert, CC BY-SA 4.0)
More on edge computing Understanding edge computing Why Linux is critical to edge computing eBook: Running Kubernetes on your Raspberry Pi Download now: The automated enterprise eBook eBook: A practical guide to home automation using open source tools eBook: 7 examples of automation on the edge The latest on edge The Matter standardRecently, a new standard released its first version, codenamed Matter. This could be a game-changer for smart home interoperability. Matter promises to connect devices from different brands and avoid purchasing a hub from each manufacturer. Matter brings direct local connectivity where before, integration between various manufacturers passed through a cloud or a concentrator hub.
However, there are questions:
- Can the millions of devices that are not Matter-compatible and already in homes still be used?
- Is there a way to have local interoperability without having a Matter-compatible device?
- Is it possible to make this interoperability open to encourage adoption and reduce barriers?
The answer to these questions is, Yes. Recently the open source Theengs platform released an ecosystem of apps, gateways, and tools to allow Bluetooth devices from different manufacturers to be integrated into a single local ecosystem.
Theengs consists of:
- App: Presents data from different sensor manufacturers on a single interface. Apps are available for phones and tablets.
- Gateway: Pushes the data to an MQTT broker with one API to be used in your favorite open source Internet of Things or smart home software. You can install gateways (as firmware and software) on a Raspberry Pi, ESP32, Linux desktop, or Windows. You can connect the gateway to an open source controller like OpenHAB, Home Assistant, or Jeedom.
- Add-on: A Home Assistant add-on to see your BLE devices automatically.
- Decoder: Decoder translates a manufacturer's information into a standard format. The decoder is the heart of Theengs.
- Tools: A host of tools to help decode devices.
Theengs supports more than 50 devices, and the list is growing. There's an excellent chance you'll find a device that fits your use case in the list, including temperature sensors, scales, pressure, and humidity.
If you have something that's not covered by Theengs yet, feel free to join in and help develop a solution or to ask for device integration in the Theengs community.
Theengs is an open source platform designed to integrate devices from different manufacturers into a single local ecosystem.
Image by:Opensource.com
Home automation Edge computing Internet of Things (IoT) What to read next Why choose open source for your home automation project This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Get started with Parseable, an open source log storage and observability platform
Log data is one of the fastest-growing segments across data storage. It's also one of the most complicated spaces. There are several products and solutions with overlapping use cases and confusing marketing.
This article looks at Parseable, a log storage and observability platform. Parseable is geared towards a better user experience, with an easy-to-deploy and use interface and a simple, cloud-native architecture. I'll also show how to set up Parseable with FluentBit to store logs.
More on Microservices Microservices cheat sheet How to explain microservices to your CEO Free eBook: Microservices vs. service-oriented architecture Free online course: Developing cloud-native applications with microservices arc… Latest microservices articles What is Parseable?Parseable is a free and open source log storage and observability platform. Written in Rust, Parseable leverages data compression, storage, and networking advances to bring a simple, efficient logging platform that just works.
Some core concepts for building Parseable are:
Indexing freeTraditionally, text search engines like Elastic have doubled as log storage platforms. This makes sense because log data must be searched to be really useful. But indexing comes at a high cost. It is CPU intensive and slows down ingestion. Also, the index data generated by these systems are of the same order of storage as the raw log data. This doubles the storage cost and increases complexity. Parseable changes this. With columnar data formats (parquet), it is possible to compress and query the log data efficiently without indexing it.
Ownership of both data and contentWith parquet as the storage format and stored in standard object storage buckets, users own their log data and have complete access to the actual content. This means users can easily use analysis tools like Spark, Presto, or TensorFlow to extract more value from the data. This feature is extremely powerful, opening up new avenues of data analysis.
Fluid schemaLogs are generally semi-structured by nature, and they're ever-evolving. For example, a developer may start with a log schema like this:
{"Status": "Ready",
"Application": "Example"
}
But as more information is collected, the log schema may evolve to:
{"Status": "Ready",
"Application": {
"UserID": "3187F492-8449-4486-A2A0-015AE34F1D09",
"Name": "Example"
}
}
Engineering and SRE teams regularly face schema-related issues. Parseable solves this with a fluid schema approach that lets users change the schema on the fly.
Simple ingestionThe current ingestion mechanism for logging platforms is quite convoluted, with several available protocols and connectors. Parseable aims to make log ingestion as easy as possible. The result is you can use HTTP POST calls to send logs to Parseable. No complicated SDKs are required.
What if you want to use a logging agent like FluentBit, Vector, LogStash, or others? Almost all the major log collectors support HTTP, so Parseable is already compatible with your favorite log collection agent.
Get startedYou can use a Docker image to try out Parseable. This image shows Parseable in demo mode, using publicly-accessible object storage.
$ cat << EOF > parseable-envP_S3_URL=https://minio.parseable.io:9000
P_S3_ACCESS_KEY=minioadmin
P_S3_SECRET_KEY=minioadmin
P_S3_REGION=us-east-1
P_S3_BUCKET=parseable
P_LOCAL_STORAGE=/data
P_USERNAME=parseable
P_PASSWORD=parseable
EOF
$ mkdir -p /tmp/data
$ docker run \
-p 8000:8000 \
--env-file parseable-env \
-v /tmp/data:/data \
parseable/parseable:latest
Log in to the Parseable UI using the credentials passed here (that's parseable and parseable.) The demo already contains some data because Parseable is pointing to the publicly-open bucket.
Make sure to change the bucket and credentials to your object storage instance before sending any data to Parseable.
Refer to the documentation to understand how Parseable works and how to ingest logs.
Set up FluentBit to send logs to ParseableYou can use a Docker compose file to configure both Parseable and FluentBit, making it easier to set up and tear down as needed.
First, save this file as fluent-bit.conf in a directory. The file is the configuration used to send data to Parseable.
[SERVICE]Flush 5
Daemon Off
Log_Level debug
[INPUT]
Name dummy
Tag dummy
[OUTPUT]
Name http
Match *
Host parseable
http_User parseable
http_Passwd parseable
format json
Port 8000
Header X-P-META-meta1 value1
Header X-P-TAG-tag1 value1
URI /api/v1/logstream/fluentbit1
Json_date_key timestamp
Json_date_format iso8601
Now save the following file as docker-compose.yaml in the same directory as above:
version: "3.7"services:
fluent-bit:
image: fluent/fluent-bit
volumes:
- ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
depends_on:
- parseable
parseable:
image: parseable/parseable
ports:
- "8000:8000"
environment:
- P_S3_URL=https://minio.parseable.io:9000
- P_S3_ACCESS_KEY=minioadmin
- P_S3_SECRET_KEY=minioadmin
- P_S3_REGION=us-east-1
- P_S3_BUCKET=parseable
- P_LOCAL_STORAGE=/tmp/data
- P_USERNAME=parseable
- P_PASSWORD=parseable
The docker-compose.yaml refers to the fluent-bit.conf file and passes it to the FluentBit container as the configuration file.
Parseable is deployed with the default configuration (as in the above Docker setup). You can observe the data FluentBit container sent to Parseable in the Parseable Console running at http://localhost:8000.
Wrap upIn this article, you've taken your first look at Parseable, the open source log storage and analysis platform built in Rust. A single Docker command gets you started with Parseable so you can experience the UI and establish FluentBit as a data source. If you think this looks too easy, then it's probably time to try Parseable!
Written in Rust, Parseable leverages data compression, storage, and networking advances to bring a simple, efficient logging platform that just works.
Image by:Opensource.com
Rust Microservices Cloud What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Transfer files and folders from Windows to Linux with WinSCP
Sometimes you need to transfer files over a network. There are lots of file sharing services out there, but most require that you send your file to the Internet. This seems like a long way to go (not to mention the privacy concerns) when two computers are right beside each other, or at least in the same building. The open source WinSCP utility makes it quick and easy to transfer a file or a folder of files over the network from your Windows computer to your Linux computer.
More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles IP addressBefore you can make the transfer, you must know the IP address or fully-qualified domain name of the destination computer. Assuming it's a computer on your same network, and that you're not running a DNS server to resolve computer names, you can find the destination IP address using the ip command on the Linux machine:
[linux]$ ip addr show | grep 'inet 'inet 127.0.0.1/8 scope host lo
inet 192.168.1.23/24 brd 10.0.1.255 scope global noprefixroute eth0
In all cases, 127.0.0.1 is a loopback address that the computer uses only to talk to itself, so in this example the correct address is 192.168.1.23. On your system, the IP address is likely to be different. If you're not sure which is which, you can try each one in succession until you get the right one (and then write it down somewhere!)
Alternatively, you can look in your router's settings, which list all addresses assigned over DHCP.
Firewalls and serversThe WinSCP command uses the OpenSSH protocol, so your Linux computer must be running the OpenSSH server software, and its firewall must allow SSH traffic.
If you're not sure whether your Linux machine is running SSH, then run this command on the Linux machine:
[linux]$ sudo systemctl enable --now sshdTo ensure your firewall allows SSH traffic, run this command:
[linux]$ sudo firewall-cmd --add-service ssh --permanentFor more information on firewalls on Linux, read Make Linux stronger with firewalls.
Using WinSCPWinSCP is an open source SSH file transfer application for Microsoft Windows. To use it, you first must download and install it.
Once you're installed it, open WinSCP and select the SCP option in the File Protocol field.
Add the IP address or DNS name of your Linux computer in the Host name field, and enter 22 in the Port number field. Enter you user name and password for the Linux computer, and then click the Login button at the bottom of the WinSCP window.
Image by:(Paul Laubscher, CC BY-SA 4.0)
Verify that you are authenticated to the Linux computer. Upon success, your Linux computer's IP address or DNS name appears at the top of the window.
Image by:(Paul Laubscher, CC BY-SA 4.0)
Now you can drag and drop a file (I used winscp-test.txt as an example) from the left Windows pane to the destination Linux computer pane on the right, and the file transfers.
Image by:(Paul Laubscher, CC BY-SA 4.0)
Alternatively, you can right-click on a file in the left pane and upload it to the remote destination in the right pane.
Image by:(Paul Laubscher, CC BY-SA 4.0)
Verify the copyOpen a Linux terminal and use the ls command to view the transferred winscp-test.txt file. In my example, it appears in my home directory, /_home_/sysadmin.
$ lsDesktop
Documents
Downloads
Music
Pictures
pscp-test.txt
[...]
You've successfully transferred a file from a Windows computer to a Linux computer over the network!
Of course, you can use the same technique as above to transfer files and folders from a Linux computer to a Windows computer.
Remote copyingWith the power of the open source WinSCP application, you have access to any computer in your house or workplace, to servers you have accounts on, and even mobile, edge, and Internet of Things devices. Use this great tool to transfer files as easily as you would copy a file from one local directory to another!
If you're looking for a way to quickly transfer files from your Windows computer to your Linux computer, then the open source WinSCP utility makes it easy to transfer a file or a folder of files over the network.
Image by:Opensource.com
Linux Windows What to read next Transfer files and folders from Windows to Linux with PSCP This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.20 technology horror stories about learning the hard way
Halloween will be here before you know it! This fun, over-the-top holiday is a great time to ponder the mortal fears of the developer in each of us. What haunts you the most, in the quiet moments just before your code starts to run?
Getting into the spirit of Halloween, I asked some Opensource.com writers: What's the scariest code you've seen or written?
More for sysadmins Enable Sysadmin blog The Automated Enterprise: A guide to managing IT with automation eBook: Ansible automation for Sysadmins Tales from the field: A system administrator's guide to IT automation eBook: A guide to Kubernetes for SREs and sysadmins Latest sysadmin articles Bad permissionsI was responsible for a server, and I FTP'd something up. There were some funky things displaying, so I thought some permissions needed to be changed.
Needless to say, I foolishly turned off read mode and took down the site. (A website is not much good when nobody can access it.)
It took me hours to fix. This was at an agency years ago when I was the sole web developer.
Shambling HTMLI took down a client's website, who was an author on the Wall Street Journal bestseller list at the time, because the original WordPress default theme had an update available.
His developer had hardcoded HTML into the theme instead of creating a child theme. I ran the update.
This was in the days when folks didn't have nightly backups easily, so I spent hours on the phone with the hosting provider. Things like staging, child themes, nightly backups, or manual backups, are all now normal things, as well as the ability to auto-update and manually roll back. Not so in that era.
Not-so-secret keyI think many of us have seen a secret key in public code before. Or another favorite: A friend of mine sending emails to 100,000 users from the dev server.
Unix mix-upThis is a Unix story. It's fixed in Linux today.
A day before I was going to give an important demo of a new component to management, I had to refresh my code (this was way before Git existed.) I went to my home directory, found the project directory, and deleted everything. Unfortunately, in that flavor of Unix, this command followed symbolic links, and I had a link to the latest version of the code (not all was on the source code system as it was still in the testing phase).
A day later, there was a network problem in the building, so the demo was delayed for a day, and we managed to recover. It was more than three decades ago. Even now I have no clue whether the network problem was a coincidence or an attempt of our sysadmin to save us (if so, it worked!)
ImperativeSeeing !important; all over a CSS file instead of proper use of specificity.
I once had to override and customize almost all of a WordPress theme's CSS because the owner of the site wouldn't budge on getting a new theme that was closer to the design he wanted.
That same theme was last updated by the developer in 2018, and the website is still using it.
MisquotedIn a previous role, my predecessor misquoted the lyrics to Journey's "Any Way You Want It" in a code comment.
The ghost of Algol68Algol68's complexity, back in the late 1960s and early 1970s, frightened away many influential people, including Niklaus Wirth. The most common complaint I can recall back then was along the lines of "who could write a compiler for such a complicated beast?" And yet many people did. Moreover, many of the concepts developed or at least formalized as Algol68 appeared in later languages, notably in C and the Bourne shell (thanks to Steve Bourne).
Some of Algol68's concepts have not aged well. The concept of I/O dealing with "books" and "chapters," and so on, is a bit weird today. Leaving things like character sets to the implementation seems pretty old-fashioned.
But some are, or should be, tremendously relevant today, such as expressions that yield a value, strong typing (types in Algol68 are called "modes"), heap memory and garbage collection, definition and overloading of operators, and more.
Sticking with the Hallowe'en theme, both tricks and treats.
Algol68 is a language that merits study, if for no other reason than to see where so many of modern computing's ideas came from, and to see how many have been lost along the way.
Passwords exposedI was doing a tech audit for an incoming support client, and the previous developer put passwords in plain text throughout the full theme, and used horrible ways to connect to a remote database. Their composer file was also ghoulishly bloated. It took five minutes every time I tried to get the site up and running locally. Outdated dependencies, repos I could not access, the list goes on.
The mazeThe scariest code I ever saw was a piece of PDP-11 assembly language in the kernel of an operating system named RSTS, which nobody remembers today. Source code was on microfiche in those days, and I had followed this code path through a few twists and turns, trying to figure out what was going on. And then, I ran into this instruction:
MOV R5,PCI threw up my hands and wailed. I really did wail. People in the office thought I'd hit my head, or had a heart attack.
In those days, memory was precious and a MOV instruction used a teeny tiny bit less memory than a BR (for "branch") instruction. Copying the contents of register 5 into the program counter was really a cheap unconditional branch to the address stored in register 5. Except, I had no clue what was stored in register 5, or how to find it.
To this day, almost 40 years later, I wonder who would write code like that and how anyone could debug it.
Off by oneI work in the automation industry, where the PLCs are programmed in some pretty weird languages.
An example that haunts me is the fact that in the language ST, you can define arrays to begin at index 1. It means that the first element is at position 1, not 0. It drives me nuts when I see it.
DivergenceI took a MongoDB instance down for 40 minutes once during a stage-to-prod launch. Our staging environment had diverged from production. It was just a database configuration difference—not very exciting. But it's a good lesson to make sure your staging and prod environments are in sync!
Unearthly whispersThis is from a project that's still alive and kicking, but I've changed the code to hide the source.
for (int c = 0; y < yyy && c < ccc; y++, c++) {// some code here
}
It seems like an innocent loop at first. But maybe you're asking why there are two variables with two stop conditions and two increments. And then you realize there's only one initializer and the second variable (y) is initialized before this loop in a different code block.
When I realized this, it took me about an hour to understand why the code was written in this way, and how it's supposed to work. Obviously, there were no c comments and the variable names are meaningless (c is called c in the code and y has a bit more meaningful name, but not meaningful enough to explain to me its meaning, not even today when I understand what it does).
More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Critical dataAround 1980, I got my first job after college. I was the Assistant Computing Center Director at an engineering college in Indiana. That's a fancy title for the second-in-command of a two-person IT shop. I handled administrative computing on a PDP-11/40, with RK05 removable "pizza platter" disk drives (2.5 MB each.) Each admin office had one drive, and part of my job was to back them up, disk to disk, every week. But I got busy over that summer and skipped the Registrar's Office four weeks in a row. And then I realized the risk, so I made sure to start my monthly disk-to-tape backup.
I dismounted the Registrar's pizza platter from the 11/40 and mounted it on the 11/70, which had a 9-track tape drive, and started my backup. A few minutes later, I heard a scraping noise inside that disk drive. Yep, the heads crashed. In a few short minutes, I'd destroyed all the Registrar's data, and the then-most-recent backup, which was a four-week-old 9 track tape.
It was a, well, uncomfortable moment when I had to look the Registrar department head in the eye and tell him I had destroyed all his data.
Today, I tell new IT people you're not a pro until you've destroyed somebody's critical data, and there's no way to recover it. Remember that feeling in the pit of your stomach forever.
Angry mobA client hacked WordPress core to add features that later came out in a routine update and couldn't understand why the site kept crashing every time they attempted to update LearnDash. (They also didn't like our report that called out their poor development practices.) They basically showed us the door calling us liars and incompetents. To this day, I still have delegate access to their domains and wp-admin access to production and development of two domains.
They also, despite us sharing a link to an encrypted location for sharing access credentials, sent our logins over emails.
Don't forget to backupI've not worked much on corporate networks, so I haven't downed any servers. However, as I young person, I tried to help a person with an IT problem and somehow caused Windows 95 to crash, and had to reinstall it for free.
Another of my saddest moments as a very young Amiga user was when my save disk, containing all my files, broke due to some mechanical failure. Nowadays, I've gotten better at backing up more of my important personal files.
Programming and development Red Hat Developers Blog Programming cheat sheets Try for free: Red Hat Learning Subscription eBook: An introduction to programming with Bash Bash shell scripting cheat sheet eBook: Modernizing Enterprise Java An open source developer's guide to building applications Root of all evilI was new to Linux, and I'd just come from DOS where I used Norton Commander. Then Midnight Commander got released and I was very happy about it. It wasn't packaged for the Linux distro I used at the time (Jurix), so I compiled it myself from source, just like other software I used at that time. It worked perfectly well, and suddenly I felt more at home on Linux.
That's not the horror story.
My colleagues told me not to run Midnight Commander as root, regardless of how comforting it was. But root was easy, and it felt more like DOS, so I ignored their advice. Long story short: I accidentally removed the content of the entire /etc directory. Until that time, I'd never had to use backups, but that day I learned that backups are actually useful.
27 years later, I still remember this story, and I do regular backups.
IllusionThe worst project one agency had me "make" was a one-pager that seemed straightforward at first. I said I'd be able to hash it together with some HTML and CSS, maybe a little Javascript. But they specifically asked me not to do that. They wanted me to cut out the design and literally use CSS to position those pieces around the page. They also had me add all CSS inline, directly into the HTML file, because they literally wanted one page.
None of the text was real text.
There were no real HTML elements aside from the ones needed to position those images.
I told them that the design was simple enough that I could throw it together with actual code, but they didn't want that. They only wanted me to spend the time to cobble the pieces together and then move on to a different project. They had me make two little one-page sites like that.
It hurt my front-end soul. It was physically painful for me to do that project. It was a temp-to-perm gig, and when they offered me full-time, I politely declined.
CorruptionThe scariest things to me are memory corruptions that can occur in ANSI C99. During a screencast, I captured this (not quite) paranormal occurrence in this YouTube clip.
Image by:(Joël Krähemann, CC BY-SA 4.0)
The GtkEntry labeled file shows some random glyphs. I've double checked the code, but didn't find any issues.
The ags_export_soundcard_open_response_callback() function is a callback to the "response" event of GtkFileChooserDialog. (For the record, the tool to target this problem is valgrind.)
Image by:(Joël Krähemann, CC BY-SA 4.0)
More Python resources What is an IDE? Cheat sheet: Python 3.7 for beginners Top Python GUI frameworks Download: 7 essential PyPI libraries Red Hat Developers Latest Python articles Python fearsThe most horrific programming feature I ever saw is the access Python gives to its dict. Changing the type of an object at runtime is against my programming code of conduct.
Franken-netIn 2006, I built firewalls based on Fedora and a bunch of scripting, and persuaded a customer with a large website inside a colo center to replace a proprietary firewall with one of mine. I built it and showed up to install it at 4AM one morning. That was when I learned (the hard way) that he had a load balancer behind his firewall, but with a public IP address. The customer endured a 5-minute outage, but I reconnected everything to the original, and it all came back online.
I found a way to handle his franken-net configuration by using proxy ARP. The idea was whenever anyone from the outside world did an ARP request for the load balancer, I would answer. A few days later, I showed up at 4AM again and installed my system. This time, I knocked everything in the entire colo center offline. I had set up my proxy ARP to respond to everything, and so all traffic on the LAN eventually found me and disappeared into a black hole.
Once I realized what I'd done, I put it all back the way it was. But the damage was done. If you tried to browse your favorite website around 4AM US Central time one morning in 2006 and it didn't respond, it might have been my fault. I knocked an entire colo site offline by installing one system in a rack and turning it on.
The website operator screamed and I slunk out the door. They never invited me back to try again. That was a shame, because bridging probably would have worked.
Your horror storyWhat's your favorite technology-related horror story? Tell us in the comments (but be nice, and change project names to protect the innocent!)
Sysadmins, web designers, engineers, and programmers share their scariest experiences on the command line.
Image by:Jen Krieger, CC BY-SA 4.0
Web development Programming Opensource.com community What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 46 points Providence RIMy official role is Solution Architect at EPAM working from home in Rhode Island. My unofficial role at any organization I work for is resident Drupal fanatic; I believe strongly in contributing to the Drupal community and supporting open source in any way I can.
I’m the organizer of the Drupal Providence Meetup, an Acquia-certified Site Builder, a co-host on Talking Drupal, and a co-organizer of the New England Drupal Camp. I hold a bachelor degree in Web Management and Internet Commerce, as well as an associate degree in Web Development from Johnson & Wales University. Throughout my career I have crafted Drupal solutions for organizations like CVS Caremark, Leica Geosystems, Blue Cross Blue Shield, Marriott International, Rhode Island School of Design, and Getty Images.
When I’m not immersed in the world of Drupal, I enjoy spending time with my family, traveling, drinking craft beer, coffee, and cooking!
| Follow johnpicozzi Open Enthusiast Author 16 points Chambersburg, PACourtney is a Web Designer and Developer Advocate, WordPress Training Team co-rep, and sponsored by GoDaddy Pro. Courtney has instructed WordPress and web development in career-technical high-schools, front-end bootcamps, and adult education. She began using open source in 1999 and contributing to WordPress in 2009.
| Follow courtneyr_dev | Connect courtneyr-dev Community Member 47 points Ottawa, Ontario, CanadaMiriam is a technical lead on the WordPress team at Kanopi Studios. She is a full-stack developer, leaning more toward the back end. She loves problem-solving, diving deep into plugin development, and mentoring junior developers.
Miriam is also heavily involved in the WordPress community, speaking and organizing WordCamps and local Ottawa meetups.
Open Enthusiast 57 points Kfar Saba, IsraelJosh is a Senior Principal Software Engineer at Red Hat.
Josh has more than 30 years of experience in software development, more than 20 years in architecture positions, and 7 years of experience as an architect in the storage development industry.
https://www.linkedin.com/in/joshsalomon/
Open Enthusiast Author 4961 points Lafayette, IndianaBen Cotton is a meteorologist by training, but weather makes a great hobby. Ben works as the Fedora Program Manager at Red Hat. He co-founded a local open source meetup group, and is a member of the Open Source Initiative and a supporter of Software Freedom Conservancy. Find him on Twitter (@FunnelFiasco) or at FunnelFiasco.com.
| Follow FunnelFiasco User Attributes Alumni Open Source Sensei Moderator's Choice Award 2017 Cloud Linux Open science Python SysAdmin Creative Commons Developer Fedora Author Comment Gardener Open Organization Ambassador Contributor Club 6916 points Vancouver, CanadaSeldom without a computer of some sort since graduating from the University of British Columbia in 1978, I have been a full-time Linux user since 2005, a full-time Solaris and SunOS user from 1986 through 2005, and UNIX System V user before that.
On the technical side of things, I have spent a great deal of my career as a consultant, doing data analysis and visualization; especially spatial data analysis. I have a substantial amount of related programming experience, using C, awk, Java, Python, PostgreSQL, PostGIS and lately Groovy. I'm looking at Julia with great interest. I have also built a few desktop and web-based applications, primarily in Java and lately in Grails with lots of JavaScript on the front end and PostgreSQL as my database of choice.
Aside from that, I spend a considerable amount of time writing proposals, technical reports and - of course - stuff on https://www.opensource.com.
User Attributes Correspondent Open Sourcerer People's Choice Award 100+ Contributions Club Emerging Contributor Award 2016 Author Comment Gardener Correspondent Columnist Contributor Club 167 points MinnesotaAfter surviving multiple layoff rounds at Digital Equipment Corporation, a large computer company in its day, I started Scott Consulting in 1994. A larger firm bought Scott Consulting in 1999, just as the dot com bust devastated the IT Service industry. A glutton for punishment, I went out on my own again in late 1999 and started Infrasupport Corporation, this time with a laser focus on infrastructure and security. I accepted a job offer with Red Hat, Inc. in 2015 as a Senior Technical Account Manager.
I'm also a published author. Jerry Barkley is an IT contractor, not a superhero. But after he uncovers a cyberattack that could lead to millions dead and nobody believes his warnings, if he doesn't act then who will? Real superheroes are ordinary people who step up when called. "Virus Bomb" and "Bullseye Breach" are available everywhere books are sold. More info at https://www.dgregscott.com/books. Enjoy the fiction. Use the education.
My family and I live near St. Paul, Minnesota.
| Follow dgregscott Open Minded Author Contributor Club 1214 points D-33415 VerlStephan is a technology enthusiast who appreciates open source for the deep insight of how things work.
Stephan works as a full time support engineer in the mostly proprietary area of industrial automation software. If possible, he works on his Python-based open source projects, writing articles, or driving motorbike.
I ❤ Open Source and I love working on projects that make a difference. Most of my work has been on projects at the intersection of open source communities, design systems, and engineering.
I work at Red Hat on the Red Hat Design System team, building web components.
I’ve been involved in the Drupal and Linux communities for 15 years, organizing Drupal events, LinuxFest Northwest, and local user groups in that time.
| Follow DyanneNova Community Member 137 points SwedenHello my name is Richard and I’m an intermediate Linux user diagnosed with ADHD and
Asperger's.
On a daily basis I use Linux for java programming, productivity and gaming.
I’m also a trained teacher, male, 39yrs of age, living in Sweden. I first started using Linux in late 90s. One of the first distros I installed was Redhat due to it's ease of use.
Today I mostly use Ubuntu and Manjaro.
I'm among other things interested in how Linux and open source software can be made more accessible to people with conditions like ADHD, Asperger's and Dyslexia.
mind.
I use accessibility software due to being diagnosed with Asperger's and ADHD.
I mostly use speech synthesis to find spelling errors and calendar software with accommodations.
I can be reached at:
rikardgn@gmail.com
Peter is an engineer working as open source evangelist at Balabit (a One Identity business), the company that developed syslog-ng. He assists distributions to maintain the syslog-ng package, follows bug trackers, helps users and talks regularly about sudo and syslog-ng at conferences (SCALE, All Things Open, FOSDEM, LOADays, and others). In his limited free time he is interested in non-x86 architectures, and works on one of his PPC or ARM machines.
| Follow PCzanik Open Source Evangelist People's Choice Award Open hardware SysAdmin Android CentOS Fedora DevOps Linux Author Contributor Club 16 pointsI’m a WordPress developer who has a keen interest in accessibility. #a11y I’m also a technical writer and I share my WordPress and coding experiences on my blog.
| Follow rachelrvasquez Community Member 173 pointsFree software enthusiast with a strong knowledge about the C programming language. I don't fear any code complexity as long it is written in a simple manner.
As developer of Advanced Gtk+ Sequencer I know how challenging multi-threaded applications can be and with it we have a great basis for future demands.
my personal website
10 universal steps for open source code review
Have you ever found yourself in a situation where you needed to do a code review but didn't fully understand the project? Maybe you did not review it to avoid looking like you didn't know what you were doing.
This article assures you that there's a better way. You don't need to know everything to provide a code review. In fact, based on my experience, that's quite common.
I remember when I joined Red Hat as an intern and was asked to help with code reviews. We used a system of +1 or -1 votes, and I was initially very hesitant to weigh in. I found myself asking whether when I gave a +1 on a change but then someone else voted -1, would I look foolish?
What does happen if someone votes -1 on a change you've vote +1? The answer is nothing! You might have missed a detail that the other person noticed. It's not the end of the world. That's why we have this voting system. Like the rest of open source, merging code is a collaborative effort.
Lately, I've been so inundated with code reviews that I can hardly keep up with them. I also noticed that the number of contributors doing these reviews steadily decreased.
For this reason, I'm writing about my point of view on writing a code review. In this article, I'll share some helpful tips and tricks. I'll show you a few questions you should ask yourself and a few ideas of what to look for when doing a code review.
What is the purpose of a code review?Have you ever written a really simple patch? Something you think is so trivial that it doesn't require a review? Maybe you merged it straight away. Later, it turns out there was a mistake, something obvious or silly, like the wrong indentation or a few duplicated lines of code instead of a function call (yes, I'm speaking from experience!).
A code review by someone else would have caught these things.
The point of a code review is to bring a fresh pair of eyes with a new perspective on the problem you're trying to solve. That new context is exactly the reason a code review is crucial.
You may think that you must be an expert in the language to review someone else's code, the project, or both. Here's a secret all code reviewers want you to know: That's wrong! You don't need to fully understand the project or the language to provide a fresh perspective on a change. There's a universal process of code review.
The universal process of a code reviewHere's my process for code review, grouped into a couple of points. The process provides questions I ask myself to help me focus on a code change and its consequences. You don't need to go in this specific order. If there's a step, you can't execute for any reason, just move to another step.
1. Understand the change, what it's trying to solve, and whyThe explanation of why the change is needed and any relevant context should be in the commit message. If it isn't, request it and feel free to -1 until it's provided.
Is it something that needs to be solved? Is it something the project should focus on, or is it completely out of scope?
2. How would you implement the solution? Would it be different?At this point, you know what the code change is about. How would you have done it? Think about this before reviewing the change in detail. If the solution you have in mind is different from the one you're reviewing, and you think it's better, bring that up in the review. You don't need to -1 it; just ask why the author didn't go in this direction and see how the discussion evolves.
3. Run the code with and without the changeI usually put a few breakpoints into the code, run it, and inspect how the new code interacts with the rest.
If you can't run the whole code, try to copy the function containing the new code to a new local file, simulate the input data, and run that. This is helpful when you either don't know how to run the whole project or when it requires a specific environment to which you don't have access.
4. Can the new code break anything?I mean, really anything. Think about the consequences.
In the case of a new command-line option, will it always be accepted by the target?
Can a situation occur when the option wouldn't be accepted or when it could conflict with something?
Maybe it's a new import. Is the new library, and possibly a new dependency, available in the older releases or systems you ship the project for?
What about security? Is the new dependency safe to use? The least you can do is run a quick Internet search to find out. Also, look for warnings in the console log. Sometimes there are more secure methods within the same library.
5. Is the code effective?You've determined that the proposed solution is probably correct. Now it's time to check the code itself, its effectiveness, and its necessity.
Check the style of the new code. Does it match the style of the project? Any open source project has (or should have) a document informing (new) contributors about the styles and good practices the project follows.
For instance, every project in the OpenStack community has a HACKING.rst file. There's often also a guide for new contributors with all the must-know information.
6. Check that all new variables and imports are usedOften, there have been many iterations of the code you're reviewing, and sometimes the final version is very different from when it started. It's easy to forget an import or a new variable that was needed in a former version of the new code. Automation usually checks these things using linting tools like flake8 in the case of Python code.
Can you rewrite the code without declaring new variables? Well, usually, yes, but the question is whether it's better that way. Does it bring any benefit? The goal isn't to create as many one-liners as possible. The goal is to write code that is both efficient and easy to read.
More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources 7. Are the new functions or methods necessary?Is there a similar function that can be reused somewhere in the project? It's always worth helping to avoid reinventing the wheel and re-implementing logic that's already been defined.
8. Are there unit tests?If the patch adds a new function or new logic in a function, it should also include new unit tests for that. It's always better when the author of a new function also writes unit tests for it.
9. Verify refactoringIf the commit refactors existing code (it renames a variable, changes variable scope, changes the footprint of a function by adding or removing arguments, or removes something), ask yourself:
- Can this be removed? Will it affect the stable branch?
- Are all the occurrences deleted?
You can use the grep command to find out. You wouldn't believe how many times I've voted -1 just because of this. This is a simple mistake that anyone can make, but that also means anyone can uncover it.
The owner of the commit can easily overlook these things, which is totally understandable. It's happened to me many times too. I'd finally figured out the root of the problem I'd been fixing, so I was in a rush to propose the review, and then I forgot to check the whole repo.
Apart from the project's repository, sometimes it's also necessary to check other code consumers. If some other project imports this one, they may need refactoring, too. In the OpenStack community, we have a tool that searches across every community project.
10. Does project documentation need to be modified?Again, you can use the grep command to check whether the project documentation mentions anything related to the code change. Apply common sense to determine whether a change needs to be documented for end users or it's just an internal change that doesn't affect user experience.
Bonus tip: Be considerateBe considerate, precise, and descriptive if you make a suggestion or comment on something after you've reviewed the new code. Ask questions if you don't understand something. If you think the code is wrong, explain why you think so. Remember, the author can't fix it if they don't know what's broken.
Final wordsThe only bad review is no review. By reviewing and voting, you provide your point of view and vote only for that. Nobody expects you to give the final yes or no (unless you're a core maintainer!), but the voting system allows you to provide your perspective and opinion. A patch owner will be glad you did it, trust me.
Can you think of any other steps for a good review? Do you have any special technique different from mine? Let us all know in the comments!
Code review doesn't have to be scary when you follow this universal process.
Image by:kris krüg
Community management Programming What to read next What makes a good code review in DevOps? This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Set up a Matrix to Discord bot
Matrix is a popular open source chat application that makes it easy to chat securely with people all over the world. Similarly, Discord is a non-open source chat application that's also popular with many online communities. Discord, like Matrix, provides a chat client for all major platforms both mobile and desktop, so it's perfectly usable on Linux. However, it's not open source and so, given the choice, you might prefer to use Matrix. The good news is that when not given a choice for any reason, you can also use Matrix to interface with Discord by running a Matrix-to-Discord bridge. This article shows you how to set up and run a Python Matrix bot to bridge chat between a Matrix room and a Discord channel.
RequirementsThe bot demonstrated in this article is a "non-puppeting" bridge, meaning that it just copies ingoing and outgoing messages on one platform and sends it to the other. There are more advanced modes available, but those tend to require a self-hosted Matrix instance. The procedure for setting up the bot, however, is similar in both cases, so whether you're setting up a bridge service for your self-hosted Matrix server or just a puppeting bot for public instances, I assume you have at least:
-
A Matrix account and a Matrix client such as Element.
-
A Discord account.
-
A Linux or BSD server that can run the Python3 bot. I use a Rev. 1 Raspberry Pi with just a 700mHZ processor and 256 MB RAM, running NetBSD. You can run the bot locally, if you prefer, but I find it more convenient to run it as a persistent service so I don't miss messages that happen while I'm away.
Download or clone matrix-discord-bridge.
Change into its bridge directory and install its dependencies using pip:
$ python3 -m pip install -r requirements.txtRun the bot to generate an empty configuration file:
$ python3 ./bridge.pyYou now have a file called config.json in your current directory. It contains six key and value pairs. The rest of this article demonstrates how to obtain these values, but first an overview:
The top three are for Matrix.
-
homeserver: The Matrix server you log in to
-
username: Your Matrix login name
-
password: Your Matrix password
Two are for Discord:
-
token: A bot developer token obtained from Discord.
-
discord_cmd_prefix: A character sequence you want to use as a shortcut for sending the bot commands through Discord.
And the final one is for both:
-
bridge: The Discord "channel" ID and the Matrix "room" ID that you're bridging. This can contain more than one channel and room pair, so you can use just one bot to bridge several rooms.
All you have to do to set up the Matrix side is open a Matrix account for your bot.
Next, you need the ID of the room you want to bridge to Discord. To get a room ID, right-click on the room icon in the left panel of Element and select Copy Link. In the URL you've just copied, there's a semicolon. The room ID is the part on left of the semicolon, and the home server of that room is to the right. For example, suppose this is the URL you just copied:
https://matrix.to/#/!DEADBEEFzzzzABCDEF:matrix.org?via=matrix.orgThe room ID is !DEADBEEFzzzzABCDEF and the home server is matrix.org.
You can now add your Matrix details to the config.json file. For example:
"homeserver": "https://matrix.org", "username": "@mybot:matrix.org", "password": "myBadPassword1234", "token": "", "discord_cmd_prefix": "", "bridge": { "": "!DEADBEEFzzzzABCDEF:matrix.org" } } ---- More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Get a Discord tokenAssuming you already have an account on Discord, open a web browser and navigate to discordapp.com/developers/applications. Once you've logged in, click the New Application button in the Applications tab.
Give your bot a name. For this example, I use mybot.
After you've defined a bot, click on it and find the Bot category in the menu on the left.
In the Bot panel, click the Add Bot button. Discord adds your bot to the panel, alerting you that "A wild bot has appeared!" in a message box. Under the name of your bot, there's a link to click to reveal your bot's token. Click the link and copy the token into your config file.
"token": "07c63.fb2823cG759.b20_852f337a6551bc",Set the bot commandChoose a sequence of characters you want to use to issue commands to the bot in Discord. In the instance of a simple bridge, you may not have any commands you need to issue, so this value probably doesn't actually matter. I set it to !b but I've never used it.
"discord_cmd_prefix": "!b",Add your bot to DiscordNow you must add your bot to the channel you want it to bridge.
-
Select OAuth2 from the menu on the left, and then URL Generator.
Image by:Seth Kenlon, CC BY-SA 4.0
-
In the Scopes section, select bot (and only bot). In the Bot Permissions section that appears under the Scopes section, activate all options under Text Permissions.
-
Copy the URL displayed at the bottom of the panel, in the Generated URL field.
Navigate to the URL you just copied, and add the bot to the channel.
You're done with the Discord web interface, but now there's one more configuration option you need from the Discord app.
Get the Discord channel IDIn User Settings of Discord (it's the gear icon next to your name on the desktop app), select Advanced. In the Advanced panel, activate Developer Mode.
Image by:Seth Kenlon, CC BY-SA 4.0
With developer mode active, go to the channel you want to bridge. For instance, you might want to bridge Matrix to the zombie apocalypse channel on the example Discord server. First, join the example Discord server. Then right-click on text channel #zombie apocalypse, and select Copy ID.
Image by:Seth Kenlon, CC BY-SA 4.0
Paste the channel ID into the config file as the first value for bridge. Your full config file now contains:
"homeserver": "https://matrix.org", "username": "@mybot:matrix.org", "password": "myBadPassword1234", "token": "07c63.fb2823cG759.b20_852f337a6551bc", "discord_cmd_prefix": "!b", "bridge": { "1030287944604463185": "!DEADBEEFzzzzABCDEF:matrix.org" } }BridgeYour bot is now acting as a bridge, at least in theory. Some Discord channels start new users in a muted state, and they must be granted special permissions to interact. If the Discord channel you're bridging is strictly managed, then you may need to speak to a moderator or administrator and request special permissions for your bot.
For your first attempt at setting this up, it's easiest to bridge a Discord server to a Matrix room that you control. That way, you can confirm that it works when unrestricted. After you've confirmed functionality, try adding it to a restricted channel.
Open source bridges another gapOpen source does a lot of heavy lifting, including in the integration space. It's to the credit of both Matrix and Discord that they provide a robust bot ecosystem that's easy to learn and easy to use. It's to the credit of some resourceful open source developers that the two can be bridged.
Run a Python Matrix bot to bridge chat between a Matrix room and a Discord channel.
Tools Python What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Write documentation like you develop code
Many engineers and craftspeople are particular about their tools. To do a job well, you need the best tools and the skills to use them. The best tools in software development can be very powerful when applied to other kinds of digital creation. The Docs as Code approach is a great example. Docs as Code entails writing documentation using the same tools and workflows used for developing code. Proponents of Docs as Code report that this method leads to better documentation while easing the workload of the people who write it.
Text formats and source controlThe most significant adjustment when moving from a more traditional documentation platform to the Docs as Code approach is that the content is stored in a text-based markup format. This change makes all the tools for text-based materials available for generating documentation. Whether you choose DocBook, Markdown, or another markup language, the transition from using just one tool to using a standard format and a variety of tools is a big change.
Finding tools that support your workflow is really important. Many developers use their coding editors when working on Docs as Code projects. Since they are already advanced-level users with that tool, it works well for them. Finding tooling that fits the other professionals on the team, such as technical writers, editors, information architects, and documentation product owners, may take more effort. A few options to consider:
- One of the many good markdown editors available
- Coding editors with good preview tools, which make them approachable for non-coders
- The web interfaces of popular Git hosting services, especially for occasional contributors
Once content is safely in a markup format, the project can use source control such as Git, an open source tool with many more features than most documentation platforms can claim:
-
A clear and detailed version history of who changed what and when. If you have good commit message culture, you may even be able to learn why the change was made.
-
Easy parallel change processes. Working in branches in Git means everyone can make all the changes they want to and combine them at the end.
-
Advanced collaboration and review tooling. All the source-control platforms are designed to review each change in detail and have as much discussion as needed until everyone is confident that the change can go ahead.
-
Automated quality checks such as spellchecking and link checking. This saves time and catches errors that might otherwise be missed.
Source control has many benefits. Just keep in mind that if you're new to source control, it has a learning curve. There are some excellent learning resources and articles for writers that can help. You can also let your curious documentarians find the learning materials that work for them rather than asking your engineers to teach them. (Ask me how I learned this—the hard way of course!)
Pull requests and review cyclesAll source-control platforms are designed around the concept of pull requests, sometimes also called merge requests. Someone, or some team, puts together a set of changes and then requests that the changes are pulled into the main project. In many ways, working with many changes at once is easier in documentation than in code. Changing one article in one place in documentation has fewer side effects than when you change code and find that there were several other sections depending on it.
The most powerful collaboration tool is the diff, which shows the difference between old and new versions in a way that's easy to follow. There are many versions of this tool available to make the comparison view easier to look at: side-by-side, inline, or even as rendered markdown rather than just text. Each team member can use the tool or tools that work best for them. For example, the web view is commonly used to look at a small change, but for something bigger I would want to look at it locally using vimdiff or Meld.
Review comments can be added to the change as a whole or to individual lines in the proposed change. Some projects adopt a maximum line length, called a hard wrap, or start each sentence on a new line to make it easier to attach comments to specific parts of a block of text. Further changes and comments can be added until the review process is complete and the change is accepted. Since the pull requests are shown in a queue on the repository for the project, this is a good way to show what's in progress and what needs review attention. The tools make it easy for reviewers to add their thoughts. In particular, if you are working with technical audiences it can be easier to get reviews from these folks via the tools they use daily.
Programming and development Red Hat Developers Blog Programming cheat sheets Try for free: Red Hat Learning Subscription eBook: An introduction to programming with Bash Bash shell scripting cheat sheet eBook: Modernizing Enterprise Java An open source developer's guide to building applications Continuous integration and deploymentHaving the source of your documentation available in plain text has many benefits, such as making it easy to find every occurrence of something that needs changing and using existing tools such as wc, grep, or tree to work with potentially large document sets. When you combine this with a source-control platform, even more existing tools become available, and they're all open source.
One big workflow improvement is the ability to have continuous deployment in place. This simply means that when a pull request is merged into the main project, the project is immediately and automatically deployed. If the change is good enough to be accepted into the project, it is also good enough to be live on the documentation site, helping your readers. Typically, continuous deployment is set up with either a separate automation server, such as Jenkins, or Git Hooks. Either way, the text-based markup is combined with the Docs as Code platform (usually a static site generator such as Hugo or Sphinx) to produce the documentation website, which is then deployed.
The same automation can be used before deployment to add some excellent checks to the pull requests before they are merged. On a coding project, it's common to run code linters, tests, and other quality checks that a machine can do itself. Documentation projects can get the same treatment, with tools like Vale to do prose linting and check for correct heading styles, spellings, and so on. It's also useful to add other tools here, such as a link checker to make sure all the links go somewhere valid.
Code tools for docs workflowsThe tools known and loved by engineers are very good tools, but they are useful for all sorts of other projects too. For documentation, they contribute valuable efficiency, especially when you need your documentation to be moving at the same speed as your development teams. All the tools discussed here are open source, so you can try them for yourself, deploy them for a huge global team, or anything in between. May your docs process be as smooth as any code process.
Don't want documentation to be an afterthought? Try a new approach.
Image by:WOCinTech Chat. Modified by Opensource.com. CC BY-SA 4.0
Documentation CI/CD Programming What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.Trick Lua into becoming an object-oriented language
Lua isn't an object-oriented programming language, but a scripting language utilizing C functions and a C-like syntax. However, there's a cool hack you can use within Lua code to make Lua act like an object-oriented language when you need it to be. The key is in the Lua table construct, and this article demonstrates how to use a Lua table as a stand-in for an object-oriented class.
What is object-oriented programming?The term "object-oriented" is a fancy way of describing, essentially, a templating system. Imagine you're programming an application to help users spot and log zombies during a zombie apocalypse. You're using an object-oriented language like C++, Java, or Python. You need to create code objects that represent different types of zombies so the user can drag them around and arrange them on a map of the city. Of course a zombie can be any number of things: dormant, slow, fast, hungry, ravenous, and so on. That's just textual data, which computers are good at tracking, and based on that data you could even assign the virtual "object" a graphic so your user can identify which general type of zombie each widget represents.
You have a few options for how you can resolve this requirement for your application:
-
Force your users to learn how to code so they can program their own zombies into your application
-
Spend the rest of your life programming every possible type of zombie into your application
-
Use a code template to define the attributes of a zombie object, allowing your users to create just the items they need, based on what zombie they've actually spotted
Obviously, the only realistic option is the final one, and it's done with a programming construct called a class. Here's what a class might look like (this example happens to be Java code, but the concept is the same across all object-oriented languages):
public class Zombie {int height;
int weight;
String speed;
String location;
} Programming and development Red Hat Developers Blog Programming cheat sheets Try for free: Red Hat Learning Subscription eBook: An introduction to programming with Bash Bash shell scripting cheat sheet eBook: Modernizing Enterprise Java An open source developer's guide to building applications
Whether or not you understand the code, you can probably tell that this is essentially a template. It's declaring that when a virtual "object" is created to represent a zombie, the programming language assigns that object four attributes (two integers representing height and weight, and two words representing the movement speed and physical location). When the user clicks the (imaginary) Add item button in the (imaginary) application, this class is used as a template (in programming, they say "a new instance" of the class has been created) to assign values entered by the user. Infinite zombies for the price of just one class. That's one of the powers of object-oriented programming.
Lua tablesIn Lua, the table is a data type that implements an associative array. You can think of a table in Lua as a database. It's a store of indexed information that you can recall by using a special syntax. Here's a very simple example:
example = {} example.greeting = "hello" example.space = " " example.subject = "world" print(example.greeting .. example.space .. example.subject)Run the example code to see the results:
$ lua ./example.lua hello worldAs you can tell from the sample code, a table is essentially a bunch of keys and values kept within a single collection (a "table").
Lua metatableA metatable is a table that serves as a template for a table. You can designate any table as a metatable, and then treat it much as you would a class in any other language.
Here's a metatable to define a zombie:
Zombie = {} function Zombie.init(h,w,s,l) local self = setmetatable({}, Zombie) self.height = h self.weight = w self.speed = s self.location = l return self end -- use the metatable function setup() z1 = Zombie.init(176,50,'slow','Forbes & Murray Avenue') end function run() print(z1.location .. ": " .. z1.height .. " cm, " .. z1.weight .. " kg, " .. z1.speed) end setup() run()To differentiate my metatable from a normal table, I capitalize the first letter of its name. That's not required, but I find it a useful convention.
Here's the results of the sample code:
$ lua ./zombie.lua Forbes & Murray Avenue: 176 cm, 50 kg, slowThis demonstration doesn't entirely do metatables justice, because the sample code creates just a single object with no interaction from the user. To use a metatable to satisfy the issue of creating infinite items from just one metatable, you would instead code a user input function (using Lua's io.read() function) asking the user to provide the details of the zombie they spotted. You'd probably even code a user interface with a "New sighting" button, or something like that. That's beyond the scope of this article, though, and would only complicate the example code.
Creating a metatableThe important thing to remember is that to create a metatable, you use this syntax:
Example = {} function Example.init(args) local self = setmetatable({}, Example) self.key = value return self endUsing a metatableTo use a metatable once it's been created, use this syntax:
my_instance = Example.init("required args")Object-oriented LuaLua isn't object-oriented, but its table mechanism makes handling and tracking and sorting data as simple as it possibly can be. It's no exaggeration to say that once you're comfortable with tables in Lua, you can be confident that you know at least half of the language. You can use tables (and, by extension, metatables) as arrays, maps, convenient variable organization, close-enough classes, and more.
Lua isn't an object-oriented programming language. Here's a hack to use a Lua table as a stand-in for an object-oriented class.
Image by:kris krüg
Programming What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.